ConcurrentRebalancersTest.TwoConcurrentRebalancers/1: Unrecognized error type. Please see the error log for more information.

Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN      ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 20:45:27.182835 19722 test_util.cc:268] Using random seed: -1361366570
I20240502 20:45:27.190690 19722 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:27.190769 19722 ts_itest-base.cc:113] --------------
I20240502 20:45:27.190793 19722 ts_itest-base.cc:114] 6 tablet servers
I20240502 20:45:27.190811 19722 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:45:27.190830 19722 ts_itest-base.cc:116] --------------
2024-05-02T20:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:27Z Disabled control of system clock
I20240502 20:45:27.202517 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.190:34537
--webserver_interface=127.19.66.190
--webserver_port=0
--builtin_ntp_servers=127.19.66.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.190:34537
--location_mapping_cmd=/tmp/dist-test-taskF51BWD/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 20:45:27.327703 19734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:27.328095 19734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:27.328174 19734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:27.331558 19734 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:27.331660 19734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:27.331708 19734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:27.331748 19734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:27.331790 19734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:27.336694 19734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskF51BWD/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.19.66.190:34537
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.190:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.66.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.19734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:27.337929 19734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:27.339349 19734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:27.349510 19742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:27.349717 19739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:27.350253 19740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:27.351315 19734 server_base.cc:1034] running on GCE node
I20240502 20:45:27.351989 19734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:27.353629 19734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:27.354903 19734 hybrid_clock.cc:648] HybridClock initialized: now 1714682727354876 us; error 47 us; skew 500 ppm
I20240502 20:45:27.357533 19734 webserver.cc:460] Webserver started at http://127.19.66.190:45729/ using document root <none> and password file <none>
I20240502 20:45:27.358165 19734 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:27.358237 19734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:27.358456 19734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:27.360924 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "62a189e5535d4595bef85381b56a39fb"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-gqqg"
I20240502 20:45:27.361686 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "62a189e5535d4595bef85381b56a39fb"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-gqqg"
I20240502 20:45:27.366165 19734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 20:45:27.370150 19748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:27.372109 19734 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 20:45:27.372380 19734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
uuid: "62a189e5535d4595bef85381b56a39fb"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-gqqg"
I20240502 20:45:27.372536 19734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:27.386874 19734 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:27.387830 19734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:27.388025 19734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:27.398663 19734 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.190:34537
I20240502 20:45:27.398669 19802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.190:34537 every 8 connection(s)
I20240502 20:45:27.399886 19734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:45:27.403057 19803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:27.409418 19803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb: Bootstrap starting.
I20240502 20:45:27.410099 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 19734
I20240502 20:45:27.410264 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:45:27.413393 19803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:27.414676 19803 log.cc:824] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:27.417007 19803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb: No bootstrap required, opened a new log
I20240502 20:45:27.421905 19803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a189e5535d4595bef85381b56a39fb" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 34537 } }
I20240502 20:45:27.422147 19803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:27.422194 19803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62a189e5535d4595bef85381b56a39fb, State: Initialized, Role: FOLLOWER
I20240502 20:45:27.422708 19803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a189e5535d4595bef85381b56a39fb" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 34537 } }
I20240502 20:45:27.422884 19803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:27.423007 19803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:27.423204 19803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:27.424611 19803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a189e5535d4595bef85381b56a39fb" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 34537 } }
I20240502 20:45:27.425007 19803 leader_election.cc:302] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62a189e5535d4595bef85381b56a39fb; no voters: 
I20240502 20:45:27.425395 19803 leader_election.cc:288] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 20:45:27.426074 19808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:27.426376 19808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [term 1 LEADER]: Becoming Leader. State: Replica: 62a189e5535d4595bef85381b56a39fb, State: Running, Role: LEADER
I20240502 20:45:27.426759 19808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a189e5535d4595bef85381b56a39fb" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 34537 } }
I20240502 20:45:27.427014 19803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:27.428360 19809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62a189e5535d4595bef85381b56a39fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a189e5535d4595bef85381b56a39fb" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 34537 } } }
I20240502 20:45:27.428548 19809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:27.428982 19810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62a189e5535d4595bef85381b56a39fb. Latest consensus state: current_term: 1 leader_uuid: "62a189e5535d4595bef85381b56a39fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a189e5535d4595bef85381b56a39fb" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 34537 } } }
I20240502 20:45:27.429198 19810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:27.429913 19817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:27.433641 19817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:27.440280 19817 catalog_manager.cc:1347] Generated new cluster ID: bb2e866ecf984fb7b8efa383f7570161
I20240502 20:45:27.440393 19817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:27.456156 19817 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:27.457470 19817 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:27.472380 19817 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 62a189e5535d4595bef85381b56a39fb: Generated new TSK 0
I20240502 20:45:27.473778 19817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:27.484791 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.129:0
--local_ip_for_outbound_sockets=127.19.66.129
--webserver_interface=127.19.66.129
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--builtin_ntp_servers=127.19.66.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:27.635848 19827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:27.636224 19827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:27.636325 19827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:27.639784 19827 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:27.639892 19827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:27.640035 19827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.129
I20240502 20:45:27.644007 19827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.66.129
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--never_fsync=true
--heap_profile_path=/tmp/kudu.19827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.129
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:27.645140 19827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:27.646446 19827 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:27.656607 19835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:27.657040 19832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:27.657972 19833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:27.658798 19827 server_base.cc:1034] running on GCE node
I20240502 20:45:27.659271 19827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:27.660440 19827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:27.661707 19827 hybrid_clock.cc:648] HybridClock initialized: now 1714682727661684 us; error 42 us; skew 500 ppm
I20240502 20:45:27.664366 19827 webserver.cc:460] Webserver started at http://127.19.66.129:44765/ using document root <none> and password file <none>
I20240502 20:45:27.665001 19827 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:27.665098 19827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:27.665328 19827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:27.667182 19827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "91942b13253244aab62c2a52e1fdd827"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-gqqg"
I20240502 20:45:27.667763 19827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "91942b13253244aab62c2a52e1fdd827"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-gqqg"
I20240502 20:45:27.671727 19827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.002s
I20240502 20:45:27.675069 19842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:27.676342 19827 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:27.676497 19827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "91942b13253244aab62c2a52e1fdd827"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-gqqg"
I20240502 20:45:27.676605 19827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:27.698181 19827 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:27.699220 19827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:27.699433 19827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:27.700035 19827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:27.701120 19827 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:27.701174 19827 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:27.701244 19827 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:27.701278 19827 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:27.715190 19827 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.129:37121
I20240502 20:45:27.715332 19957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.129:37121 every 8 connection(s)
I20240502 20:45:27.716334 19827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:45:27.723426 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 19827
I20240502 20:45:27.723541 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:45:27.725941 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.130:0
--local_ip_for_outbound_sockets=127.19.66.130
--webserver_interface=127.19.66.130
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--builtin_ntp_servers=127.19.66.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:27.727958 19958 heartbeater.cc:347] Connected to a master server at 127.19.66.190:34537
I20240502 20:45:27.728222 19958 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:27.728876 19958 heartbeater.cc:511] Master 127.19.66.190:34537 requested a full tablet report, sending...
I20240502 20:45:27.774314 19767 ts_manager.cc:194] Registered new tserver with Master: 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:27.776791 19767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.129:54725
W20240502 20:45:27.859510 19961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:27.859763 19961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:27.859812 19961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:27.863222 19961 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:27.863310 19961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:27.863389 19961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.130
I20240502 20:45:27.867501 19961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.66.130
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--never_fsync=true
--heap_profile_path=/tmp/kudu.19961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.130
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:27.869131 19961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:27.871069 19961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:27.881429 19971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:27.881919 19969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:27.882581 19968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:27.883667 19961 server_base.cc:1034] running on GCE node
I20240502 20:45:27.884156 19961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:27.884887 19961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:27.886070 19961 hybrid_clock.cc:648] HybridClock initialized: now 1714682727886044 us; error 50 us; skew 500 ppm
I20240502 20:45:27.888382 19961 webserver.cc:460] Webserver started at http://127.19.66.130:44789/ using document root <none> and password file <none>
I20240502 20:45:27.889050 19961 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:27.889160 19961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:27.889385 19961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:27.891243 19961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "951291b2e23f46a79812e404a06690d8"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-gqqg"
I20240502 20:45:27.891778 19961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "951291b2e23f46a79812e404a06690d8"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-gqqg"
I20240502 20:45:27.895577 19961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.002s
I20240502 20:45:27.898386 19977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:27.899852 19961 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 20:45:27.900040 19961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "951291b2e23f46a79812e404a06690d8"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-gqqg"
I20240502 20:45:27.900219 19961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:27.911936 19961 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:27.912760 19961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:27.912938 19961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:27.913668 19961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:27.914767 19961 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:27.914820 19961 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:27.914870 19961 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:27.914903 19961 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:27.925697 19961 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.130:34863
I20240502 20:45:27.925719 20092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.130:34863 every 8 connection(s)
I20240502 20:45:27.926815 19961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:45:27.934098 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 19961
I20240502 20:45:27.934201 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:45:27.936787 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.131:0
--local_ip_for_outbound_sockets=127.19.66.131
--webserver_interface=127.19.66.131
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--builtin_ntp_servers=127.19.66.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:27.939841 20093 heartbeater.cc:347] Connected to a master server at 127.19.66.190:34537
I20240502 20:45:27.940145 20093 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:27.940914 20093 heartbeater.cc:511] Master 127.19.66.190:34537 requested a full tablet report, sending...
I20240502 20:45:28.008366 19767 ts_manager.cc:194] Registered new tserver with Master: 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:28.009176 19767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.130:50793
W20240502 20:45:28.093096 20096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:28.093367 20096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:28.093451 20096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:28.096807 20096 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:28.096895 20096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:28.097003 20096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.131
I20240502 20:45:28.100955 20096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.66.131
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--never_fsync=true
--heap_profile_path=/tmp/kudu.20096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.131
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:28.102080 20096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:28.103325 20096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:28.112545 20102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:28.113168 20105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:28.113652 20103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:28.114420 20096 server_base.cc:1034] running on GCE node
I20240502 20:45:28.114841 20096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:28.115502 20096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:28.116691 20096 hybrid_clock.cc:648] HybridClock initialized: now 1714682728116671 us; error 32 us; skew 500 ppm
I20240502 20:45:28.118924 20096 webserver.cc:460] Webserver started at http://127.19.66.131:40547/ using document root <none> and password file <none>
I20240502 20:45:28.119524 20096 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:28.119586 20096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:28.119776 20096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:28.121748 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.122275 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.126204 20096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 20:45:28.128966 20111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.130396 20096 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:28.130538 20096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.130661 20096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:28.146847 20096 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:28.147559 20096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:28.147717 20096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:28.148290 20096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:28.149238 20096 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:28.149320 20096 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.149384 20096 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:28.149422 20096 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.159456 20096 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.131:44905
I20240502 20:45:28.159669 20224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.131:44905 every 8 connection(s)
I20240502 20:45:28.160470 20096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:45:28.165729 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 20096
I20240502 20:45:28.165861 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:45:28.168853 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.132:0
--local_ip_for_outbound_sockets=127.19.66.132
--webserver_interface=127.19.66.132
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--builtin_ntp_servers=127.19.66.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:28.172560 20225 heartbeater.cc:347] Connected to a master server at 127.19.66.190:34537
I20240502 20:45:28.172837 20225 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:28.173545 20225 heartbeater.cc:511] Master 127.19.66.190:34537 requested a full tablet report, sending...
I20240502 20:45:28.243636 19767 ts_manager.cc:194] Registered new tserver with Master: 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:28.244495 19767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.131:38389
W20240502 20:45:28.327028 20228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:28.327332 20228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:28.327404 20228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:28.330730 20228 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:28.330824 20228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:28.330932 20228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.132
I20240502 20:45:28.335078 20228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.66.132
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--never_fsync=true
--heap_profile_path=/tmp/kudu.20228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.132
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:28.336170 20228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:28.337487 20228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:28.347352 20235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:28.347651 20238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:28.348301 20236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:28.349300 20228 server_base.cc:1034] running on GCE node
I20240502 20:45:28.349758 20228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:28.350418 20228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:28.351612 20228 hybrid_clock.cc:648] HybridClock initialized: now 1714682728351584 us; error 45 us; skew 500 ppm
I20240502 20:45:28.358058 20228 webserver.cc:460] Webserver started at http://127.19.66.132:33227/ using document root <none> and password file <none>
I20240502 20:45:28.358680 20228 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:28.358745 20228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:28.358973 20228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:28.360834 20228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c302712f8dbd4fa7bed797758fd772af"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.361407 20228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c302712f8dbd4fa7bed797758fd772af"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.365285 20228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 20:45:28.368085 20244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.369551 20228 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:28.369695 20228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c302712f8dbd4fa7bed797758fd772af"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.369818 20228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:28.390398 20228 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:28.391129 20228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:28.391309 20228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:28.391920 20228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:28.392897 20228 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:28.392951 20228 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.392992 20228 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:28.393008 20228 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.403475 20228 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.132:36391
I20240502 20:45:28.403514 20365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.132:36391 every 8 connection(s)
I20240502 20:45:28.404528 20228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:28.408005 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 20228
I20240502 20:45:28.408138 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:45:28.410634 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.133:0
--local_ip_for_outbound_sockets=127.19.66.133
--webserver_interface=127.19.66.133
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--builtin_ntp_servers=127.19.66.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:28.415789 20366 heartbeater.cc:347] Connected to a master server at 127.19.66.190:34537
I20240502 20:45:28.416031 20366 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:28.416595 20366 heartbeater.cc:511] Master 127.19.66.190:34537 requested a full tablet report, sending...
I20240502 20:45:28.462074 19767 ts_manager.cc:194] Registered new tserver with Master: c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391)
I20240502 20:45:28.463120 19767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.132:38427
W20240502 20:45:28.546802 20369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:28.547057 20369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:28.547106 20369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:28.550594 20369 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:28.550675 20369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:28.550748 20369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.133
I20240502 20:45:28.554738 20369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.66.133
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--never_fsync=true
--heap_profile_path=/tmp/kudu.20369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.133
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:28.555778 20369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:28.556972 20369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:28.566495 20376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:28.566936 20378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:28.567562 20375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:28.568507 20369 server_base.cc:1034] running on GCE node
I20240502 20:45:28.568940 20369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:28.569586 20369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:28.570763 20369 hybrid_clock.cc:648] HybridClock initialized: now 1714682728570743 us; error 43 us; skew 500 ppm
I20240502 20:45:28.572862 20369 webserver.cc:460] Webserver started at http://127.19.66.133:46681/ using document root <none> and password file <none>
I20240502 20:45:28.573522 20369 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:28.573590 20369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:28.573782 20369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:28.575557 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5742a696d4c3406086b710e571e2cf0e"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.576053 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5742a696d4c3406086b710e571e2cf0e"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.579749 20369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.006s	sys 0.000s
I20240502 20:45:28.582460 20384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.583587 20369 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:28.583732 20369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5742a696d4c3406086b710e571e2cf0e"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.583837 20369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:28.612596 20369 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:28.613591 20369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:28.613864 20369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:28.614506 20369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:28.615451 20369 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:28.615504 20369 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.615571 20369 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:28.615597 20369 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.625764 20369 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.133:34211
I20240502 20:45:28.625844 20497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.133:34211 every 8 connection(s)
I20240502 20:45:28.626958 20369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:28.628530 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 20369
I20240502 20:45:28.628667 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:45:28.631278 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.134:0
--local_ip_for_outbound_sockets=127.19.66.134
--webserver_interface=127.19.66.134
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--builtin_ntp_servers=127.19.66.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:28.639477 20498 heartbeater.cc:347] Connected to a master server at 127.19.66.190:34537
I20240502 20:45:28.639743 20498 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:28.640530 20498 heartbeater.cc:511] Master 127.19.66.190:34537 requested a full tablet report, sending...
I20240502 20:45:28.681957 19767 ts_manager.cc:194] Registered new tserver with Master: 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:28.683081 19767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.133:47205
W20240502 20:45:28.757786 20501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:28.758028 20501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:28.758077 20501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:28.761298 20501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:28.761372 20501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:28.761469 20501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.134
I20240502 20:45:28.765499 20501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.19.66.134
--webserver_port=0
--tserver_master_addrs=127.19.66.190:34537
--never_fsync=true
--heap_profile_path=/tmp/kudu.20501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.134
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:28.766474 20501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:28.767650 20501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:28.776192 20510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:28.776789 20507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:28.777374 20508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:28.778254 20501 server_base.cc:1034] running on GCE node
I20240502 20:45:28.778659 20501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:28.779253 20501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:28.780187 19958 heartbeater.cc:503] Master 127.19.66.190:34537 was elected leader, sending a full tablet report...
I20240502 20:45:28.780419 20501 hybrid_clock.cc:648] HybridClock initialized: now 1714682728780399 us; error 33 us; skew 500 ppm
I20240502 20:45:28.782860 20501 webserver.cc:460] Webserver started at http://127.19.66.134:38855/ using document root <none> and password file <none>
I20240502 20:45:28.783488 20501 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:28.783555 20501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:28.783753 20501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:28.785538 20501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "fa7ea13bcb524353a03145dda1461c9e"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.786007 20501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "fa7ea13bcb524353a03145dda1461c9e"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.789574 20501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 20:45:28.791960 20516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.793146 20501 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240502 20:45:28.793274 20501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fa7ea13bcb524353a03145dda1461c9e"
format_stamp: "Formatted at 2024-05-02 20:45:28 on dist-test-slave-gqqg"
I20240502 20:45:28.793383 20501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:28.821439 20501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:28.822149 20501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:28.822332 20501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:28.822908 20501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:28.823827 20501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:28.823877 20501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.823943 20501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:28.823971 20501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:28.833652 20501 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.134:35393
I20240502 20:45:28.833673 20630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.134:35393 every 8 connection(s)
I20240502 20:45:28.834635 20501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:45:28.839867 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 20501
I20240502 20:45:28.839975 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 20:45:28.847504 20631 heartbeater.cc:347] Connected to a master server at 127.19.66.190:34537
I20240502 20:45:28.847796 20631 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:28.848448 20631 heartbeater.cc:511] Master 127.19.66.190:34537 requested a full tablet report, sending...
I20240502 20:45:28.898298 19767 ts_manager.cc:194] Registered new tserver with Master: fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:28.899737 19767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.134:48955
I20240502 20:45:28.908084 19722 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 20:45:28.922037 19722 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 20:45:29.012084 20093 heartbeater.cc:503] Master 127.19.66.190:34537 was elected leader, sending a full tablet report...
I20240502 20:45:29.247087 20225 heartbeater.cc:503] Master 127.19.66.190:34537 was elected leader, sending a full tablet report...
I20240502 20:45:29.466063 20366 heartbeater.cc:503] Master 127.19.66.190:34537 was elected leader, sending a full tablet report...
I20240502 20:45:29.685758 20498 heartbeater.cc:503] Master 127.19.66.190:34537 was elected leader, sending a full tablet report...
I20240502 20:45:29.903378 20631 heartbeater.cc:503] Master 127.19.66.190:34537 was elected leader, sending a full tablet report...
I20240502 20:45:32.675081 19767 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:32.699539 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 07a28fca0dfe49cd907752f80507e547 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7c28cce86004e60b736cdff1a760ccb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:32.699499 20565 tablet_service.cc:1467] Processing CreateTablet for tablet 07a28fca0dfe49cd907752f80507e547 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7c28cce86004e60b736cdff1a760ccb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:32.700742 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a28fca0dfe49cd907752f80507e547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:32.700742 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a28fca0dfe49cd907752f80507e547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:32.703581 20026 tablet_service.cc:1467] Processing CreateTablet for tablet 07a28fca0dfe49cd907752f80507e547 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7c28cce86004e60b736cdff1a760ccb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:32.704668 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a28fca0dfe49cd907752f80507e547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:32.704566 20157 tablet_service.cc:1467] Processing CreateTablet for tablet f8ffa426f4bd4b5fb2c92d59fa4db546 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7c28cce86004e60b736cdff1a760ccb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:32.705600 20157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ffa426f4bd4b5fb2c92d59fa4db546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:32.709409 20155 tablet_service.cc:1467] Processing CreateTablet for tablet 8be5a401b84340258861b5d5fe02bd09 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7c28cce86004e60b736cdff1a760ccb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:32.709714 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8be5a401b84340258861b5d5fe02bd09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:32.709728 20431 tablet_service.cc:1467] Processing CreateTablet for tablet f8ffa426f4bd4b5fb2c92d59fa4db546 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7c28cce86004e60b736cdff1a760ccb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:32.710654 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ffa426f4bd4b5fb2c92d59fa4db546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:32.709702 20432 tablet_service.cc:1467] Processing CreateTablet for tablet 8be5a401b84340258861b5d5fe02bd09 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7c28cce86004e60b736cdff1a760ccb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:32.711334 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8be5a401b84340258861b5d5fe02bd09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:32.712822 20656 tablet_bootstrap.cc:492] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:32.716220 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 8be5a401b84340258861b5d5fe02bd09 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7c28cce86004e60b736cdff1a760ccb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:32.717322 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8be5a401b84340258861b5d5fe02bd09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:32.721915 20659 tablet_bootstrap.cc:492] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:32.722106 19890 tablet_service.cc:1467] Processing CreateTablet for tablet f8ffa426f4bd4b5fb2c92d59fa4db546 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7c28cce86004e60b736cdff1a760ccb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:32.722478 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ffa426f4bd4b5fb2c92d59fa4db546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:32.725455 20659 tablet_bootstrap.cc:654] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:32.725456 20656 tablet_bootstrap.cc:654] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:32.726470 20656 log.cc:824] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:32.726470 20659 log.cc:824] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:32.727315 20658 tablet_bootstrap.cc:492] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:32.728964 20662 tablet_bootstrap.cc:492] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:32.729830 20658 tablet_bootstrap.cc:654] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:32.730703 20658 log.cc:824] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:32.731340 20662 tablet_bootstrap.cc:654] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:32.731559 20656 tablet_bootstrap.cc:492] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:32.731835 20656 ts_tablet_manager.cc:1367] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.019s	user 0.006s	sys 0.000s
I20240502 20:45:32.732064 20657 tablet_bootstrap.cc:492] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:32.732848 20659 tablet_bootstrap.cc:492] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:32.733121 20659 ts_tablet_manager.cc:1367] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.000s
I20240502 20:45:32.734274 20664 tablet_bootstrap.cc:492] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:32.734301 20657 tablet_bootstrap.cc:654] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:32.736533 20658 tablet_bootstrap.cc:492] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:32.736377 20656 raft_consensus.cc:348] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:32.736778 20658 ts_tablet_manager.cc:1367] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.000s
I20240502 20:45:32.736771 20664 tablet_bootstrap.cc:654] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:32.736850 20656 raft_consensus.cc:374] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:32.737020 20656 raft_consensus.cc:729] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:32.737843 20664 log.cc:824] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:32.737757 20656 consensus_queue.cc:260] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:32.737905 20659 raft_consensus.cc:348] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:32.738175 20659 raft_consensus.cc:374] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:32.738269 20659 raft_consensus.cc:729] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:32.738883 20659 consensus_queue.cc:260] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:32.739969 20659 ts_tablet_manager.cc:1398] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 20:45:32.740437 20664 tablet_bootstrap.cc:492] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:32.740659 20664 ts_tablet_manager.cc:1367] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.005s
I20240502 20:45:32.741106 20662 log.cc:824] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:32.741379 20658 raft_consensus.cc:348] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.741602 20658 raft_consensus.cc:374] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:32.741655 20658 raft_consensus.cc:729] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:32.742086 20658 consensus_queue.cc:260] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.743038 20658 ts_tablet_manager.cc:1398] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240502 20:45:32.743419 20658 tablet_bootstrap.cc:492] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:32.744735 20658 tablet_bootstrap.cc:654] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:32.745663 20657 log.cc:824] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:32.746261 20664 raft_consensus.cc:348] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:32.746598 20664 raft_consensus.cc:374] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:32.746644 20656 ts_tablet_manager.cc:1398] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20240502 20:45:32.746704 20664 raft_consensus.cc:729] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:32.747076 20658 tablet_bootstrap.cc:492] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:32.747180 20658 ts_tablet_manager.cc:1367] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:32.747334 20664 consensus_queue.cc:260] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:32.747746 20658 raft_consensus.cc:348] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.747843 20658 raft_consensus.cc:374] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:32.747888 20658 raft_consensus.cc:729] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:32.748023 20658 consensus_queue.cc:260] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.748312 20658 ts_tablet_manager.cc:1398] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:32.748395 20664 ts_tablet_manager.cc:1398] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 20:45:32.750752 20657 tablet_bootstrap.cc:492] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:32.750984 20657 ts_tablet_manager.cc:1367] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.000s
I20240502 20:45:32.751022 20662 tablet_bootstrap.cc:492] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:32.751190 20662 ts_tablet_manager.cc:1367] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.022s	user 0.005s	sys 0.000s
I20240502 20:45:32.755291 20662 raft_consensus.cc:348] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.755271 20657 raft_consensus.cc:348] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.755545 20657 raft_consensus.cc:374] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:32.755544 20662 raft_consensus.cc:374] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:32.755600 20662 raft_consensus.cc:729] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:32.755612 20657 raft_consensus.cc:729] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:32.756239 20662 consensus_queue.cc:260] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.756239 20657 consensus_queue.cc:260] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.758026 20657 ts_tablet_manager.cc:1398] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.006s
I20240502 20:45:32.758546 20657 tablet_bootstrap.cc:492] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:32.758564 20662 ts_tablet_manager.cc:1398] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20240502 20:45:32.758827 20662 tablet_bootstrap.cc:492] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:32.759945 20662 tablet_bootstrap.cc:654] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:32.760666 20657 tablet_bootstrap.cc:654] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:32.761802 20662 tablet_bootstrap.cc:492] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:32.761955 20662 ts_tablet_manager.cc:1367] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:32.762143 20657 tablet_bootstrap.cc:492] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:32.762333 20657 ts_tablet_manager.cc:1367] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:32.762822 20662 raft_consensus.cc:348] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.763015 20662 raft_consensus.cc:374] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:32.763070 20662 raft_consensus.cc:729] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:32.763236 20662 consensus_queue.cc:260] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.763532 20662 ts_tablet_manager.cc:1398] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 20:45:32.763669 20657 raft_consensus.cc:348] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.763804 20657 raft_consensus.cc:374] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:32.764075 20657 raft_consensus.cc:729] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:32.764283 20657 consensus_queue.cc:260] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.764873 20657 ts_tablet_manager.cc:1398] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:32.825335 20667 raft_consensus.cc:482] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:32.825554 20667 raft_consensus.cc:504] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:32.826766 20667 leader_election.cc:288] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:32.832118 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a28fca0dfe49cd907752f80507e547" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8" is_pre_election: true
I20240502 20:45:32.832121 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a28fca0dfe49cd907752f80507e547" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:32.832489 20046 raft_consensus.cc:2454] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:32.832489 20585 raft_consensus.cc:2454] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:32.832993 20249 leader_election.cc:302] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:32.833264 20667 raft_consensus.cc:2781] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:32.833339 20667 raft_consensus.cc:482] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:32.833380 20667 raft_consensus.cc:3037] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:32.834444 20667 raft_consensus.cc:504] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:32.834844 20667 leader_election.cc:288] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Requested vote from peers 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:32.835266 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a28fca0dfe49cd907752f80507e547" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:32.835422 20585 raft_consensus.cc:3037] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:32.836323 20585 raft_consensus.cc:2454] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:32.836649 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a28fca0dfe49cd907752f80507e547" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8"
I20240502 20:45:32.836690 20249 leader_election.cc:302] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:32.836794 20046 raft_consensus.cc:3037] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:32.836870 20667 raft_consensus.cc:2781] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:32.837113 20667 raft_consensus.cc:686] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [term 1 LEADER]: Becoming Leader. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Running, Role: LEADER
I20240502 20:45:32.837453 20667 consensus_queue.cc:237] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:32.838047 20046 raft_consensus.cc:2454] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
W20240502 20:45:32.839411 20632 tablet.cc:2369] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:32.840662 20668 raft_consensus.cc:482] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:32.840828 20668 raft_consensus.cc:504] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.841815 19766 catalog_manager.cc:5484] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af reported cstate change: term changed from 0 to 1, leader changed from <none> to c302712f8dbd4fa7bed797758fd772af (127.19.66.132). New cstate: current_term: 1 leader_uuid: "c302712f8dbd4fa7bed797758fd772af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:32.842437 20668 leader_election.cc:288] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:32.847107 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8be5a401b84340258861b5d5fe02bd09" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:32.847107 20172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8be5a401b84340258861b5d5fe02bd09" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" is_pre_election: true
I20240502 20:45:32.847456 20172 raft_consensus.cc:2454] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 0.
I20240502 20:45:32.847456 19911 raft_consensus.cc:2454] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 0.
I20240502 20:45:32.847997 20386 leader_election.cc:302] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:32.848311 20668 raft_consensus.cc:2781] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:32.848397 20668 raft_consensus.cc:482] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:32.848459 20668 raft_consensus.cc:3037] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:32.849462 20668 raft_consensus.cc:504] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.849826 20668 leader_election.cc:288] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 election: Requested vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:32.850243 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8be5a401b84340258861b5d5fe02bd09" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:32.850373 20172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8be5a401b84340258861b5d5fe02bd09" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
I20240502 20:45:32.850397 19911 raft_consensus.cc:3037] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:32.850512 20172 raft_consensus.cc:3037] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:32.851658 19911 raft_consensus.cc:2454] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 1.
I20240502 20:45:32.851658 20172 raft_consensus.cc:2454] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 1.
I20240502 20:45:32.852012 20385 leader_election.cc:302] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:32.852224 20668 raft_consensus.cc:2781] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:32.852458 20668 raft_consensus.cc:686] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [term 1 LEADER]: Becoming Leader. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Running, Role: LEADER
I20240502 20:45:32.852787 20668 consensus_queue.cc:237] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.856194 19766 catalog_manager.cc:5484] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5742a696d4c3406086b710e571e2cf0e (127.19.66.133). New cstate: current_term: 1 leader_uuid: "5742a696d4c3406086b710e571e2cf0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:32.870857 20668 raft_consensus.cc:482] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:32.870995 20668 raft_consensus.cc:504] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.871389 20668 leader_election.cc:288] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:32.871913 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffa426f4bd4b5fb2c92d59fa4db546" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:32.871912 20172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffa426f4bd4b5fb2c92d59fa4db546" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" is_pre_election: true
I20240502 20:45:32.872071 19911 raft_consensus.cc:2454] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 0.
I20240502 20:45:32.872071 20172 raft_consensus.cc:2454] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 0.
I20240502 20:45:32.872375 20386 leader_election.cc:302] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 
I20240502 20:45:32.872550 20668 raft_consensus.cc:2781] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:32.872614 20668 raft_consensus.cc:482] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:32.872664 20668 raft_consensus.cc:3037] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:32.873696 20668 raft_consensus.cc:504] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.874060 20668 leader_election.cc:288] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 election: Requested vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:32.874456 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffa426f4bd4b5fb2c92d59fa4db546" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:32.874531 20172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ffa426f4bd4b5fb2c92d59fa4db546" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
I20240502 20:45:32.874578 19911 raft_consensus.cc:3037] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:32.874684 20172 raft_consensus.cc:3037] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:32.875504 20172 raft_consensus.cc:2454] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 1.
I20240502 20:45:32.875515 19911 raft_consensus.cc:2454] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 1.
I20240502 20:45:32.875815 20385 leader_election.cc:302] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:32.875995 20668 raft_consensus.cc:2781] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:32.876078 20668 raft_consensus.cc:686] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [term 1 LEADER]: Becoming Leader. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Running, Role: LEADER
I20240502 20:45:32.876224 20668 consensus_queue.cc:237] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:32.878015 19766 catalog_manager.cc:5484] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5742a696d4c3406086b710e571e2cf0e (127.19.66.133). New cstate: current_term: 1 leader_uuid: "5742a696d4c3406086b710e571e2cf0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:32.882015 20499 tablet.cc:2369] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:32.910043 20367 tablet.cc:2369] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:32.915951 20226 tablet.cc:2369] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:32.932677 20094 tablet.cc:2369] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:32.973317 19959 tablet.cc:2369] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:33.026074 20686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:33.026417 20686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:33.030073 20686 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 20:45:33.149957 20172 raft_consensus.cc:1261] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Refusing update from remote peer 5742a696d4c3406086b710e571e2cf0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.150243 19911 raft_consensus.cc:1261] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Refusing update from remote peer 5742a696d4c3406086b710e571e2cf0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.150609 20179 raft_consensus.cc:1261] T 8be5a401b84340258861b5d5fe02bd09 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Refusing update from remote peer 5742a696d4c3406086b710e571e2cf0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.150962 20668 consensus_queue.cc:1035] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.151203 20718 consensus_queue.cc:1035] T f8ffa426f4bd4b5fb2c92d59fa4db546 P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.151521 20668 consensus_queue.cc:1035] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.150249 19910 raft_consensus.cc:1261] T 8be5a401b84340258861b5d5fe02bd09 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Refusing update from remote peer 5742a696d4c3406086b710e571e2cf0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.153255 20683 consensus_queue.cc:1035] T 8be5a401b84340258861b5d5fe02bd09 P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.153733 20585 raft_consensus.cc:1261] T 07a28fca0dfe49cd907752f80507e547 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Refusing update from remote peer c302712f8dbd4fa7bed797758fd772af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.154398 20667 consensus_queue.cc:1035] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.154572 20046 raft_consensus.cc:1261] T 07a28fca0dfe49cd907752f80507e547 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Refusing update from remote peer c302712f8dbd4fa7bed797758fd772af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.155855 20667 consensus_queue.cc:1035] T 07a28fca0dfe49cd907752f80507e547 P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.170389 20715 mvcc.cc:205] Tried to move back new op lower bound from 7023340474973925376 to 7023340473767780352. Current Snapshot: MvccSnapshot[applied={T|T < 7023340474973925376}]
I20240502 20:45:33.171994 20719 mvcc.cc:205] Tried to move back new op lower bound from 7023340474975645696 to 7023340473862447104. Current Snapshot: MvccSnapshot[applied={T|T < 7023340474975645696}]
I20240502 20:45:33.192776 20717 mvcc.cc:205] Tried to move back new op lower bound from 7023340474989834240 to 7023340473706336256. Current Snapshot: MvccSnapshot[applied={T|T < 7023340474989834240}]
I20240502 20:45:33.208498 20725 mvcc.cc:205] Tried to move back new op lower bound from 7023340474989834240 to 7023340473706336256. Current Snapshot: MvccSnapshot[applied={T|T < 7023340474989834240}]
I20240502 20:45:33.211300 20728 mvcc.cc:205] Tried to move back new op lower bound from 7023340474989834240 to 7023340473706336256. Current Snapshot: MvccSnapshot[applied={T|T < 7023340474989834240}]
I20240502 20:45:33.213150 20726 mvcc.cc:205] Tried to move back new op lower bound from 7023340474973925376 to 7023340473767780352. Current Snapshot: MvccSnapshot[applied={T|T < 7023340474973925376}]

INSERT report
    rows total: 2000
    time total: 138.256 ms
  time per row: 0.0691282 ms
I20240502 20:45:33.233917 19761 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:33.241317 20296 tablet_service.cc:1467] Processing CreateTablet for tablet df575a25f570474f8b67748701025faa (DEFAULT_TABLE table=rebalance_test_table_1 [id=06f77ab650764027b3648c3f650f13ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.241555 20026 tablet_service.cc:1467] Processing CreateTablet for tablet df575a25f570474f8b67748701025faa (DEFAULT_TABLE table=rebalance_test_table_1 [id=06f77ab650764027b3648c3f650f13ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.241688 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df575a25f570474f8b67748701025faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.241688 20565 tablet_service.cc:1467] Processing CreateTablet for tablet df575a25f570474f8b67748701025faa (DEFAULT_TABLE table=rebalance_test_table_1 [id=06f77ab650764027b3648c3f650f13ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.241865 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df575a25f570474f8b67748701025faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.241925 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df575a25f570474f8b67748701025faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.242568 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 90fd3f27ca0d4e70afc9b2e680455a88 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06f77ab650764027b3648c3f650f13ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.242846 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90fd3f27ca0d4e70afc9b2e680455a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.244153 20659 tablet_bootstrap.cc:492] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:33.244240 20656 tablet_bootstrap.cc:492] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:33.245041 20659 tablet_bootstrap.cc:654] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.245203 20658 tablet_bootstrap.cc:492] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:33.245219 20664 tablet_bootstrap.cc:492] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:33.245441 19890 tablet_service.cc:1467] Processing CreateTablet for tablet 90fd3f27ca0d4e70afc9b2e680455a88 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06f77ab650764027b3648c3f650f13ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.245738 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90fd3f27ca0d4e70afc9b2e680455a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.246129 20664 tablet_bootstrap.cc:654] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.246131 20658 tablet_bootstrap.cc:654] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.246577 20659 tablet_bootstrap.cc:492] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:33.246677 20659 ts_tablet_manager.cc:1367] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:33.247089 20656 tablet_bootstrap.cc:654] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.247262 20659 raft_consensus.cc:348] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:33.247372 20659 raft_consensus.cc:374] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.247393 20659 raft_consensus.cc:729] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.247512 20659 consensus_queue.cc:260] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:33.247794 20659 ts_tablet_manager.cc:1398] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:33.248577 20658 tablet_bootstrap.cc:492] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:33.248659 20662 tablet_bootstrap.cc:492] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:33.248679 20658 ts_tablet_manager.cc:1367] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:33.248689 20157 tablet_service.cc:1467] Processing CreateTablet for tablet 90fd3f27ca0d4e70afc9b2e680455a88 (DEFAULT_TABLE table=rebalance_test_table_1 [id=06f77ab650764027b3648c3f650f13ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.248966 20157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90fd3f27ca0d4e70afc9b2e680455a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.249174 20658 raft_consensus.cc:348] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.249280 20658 raft_consensus.cc:374] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.249310 20658 raft_consensus.cc:729] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.249353 20026 tablet_service.cc:1467] Processing CreateTablet for tablet 4134ba0b3d324d69b69201205a09725d (DEFAULT_TABLE table=rebalance_test_table_1 [id=06f77ab650764027b3648c3f650f13ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.249435 20658 consensus_queue.cc:260] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.249537 20662 tablet_bootstrap.cc:654] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.249625 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4134ba0b3d324d69b69201205a09725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.249728 20658 ts_tablet_manager.cc:1398] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:33.250440 20565 tablet_service.cc:1467] Processing CreateTablet for tablet 4134ba0b3d324d69b69201205a09725d (DEFAULT_TABLE table=rebalance_test_table_1 [id=06f77ab650764027b3648c3f650f13ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.250737 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4134ba0b3d324d69b69201205a09725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.252177 20657 tablet_bootstrap.cc:492] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:33.252424 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 4134ba0b3d324d69b69201205a09725d (DEFAULT_TABLE table=rebalance_test_table_1 [id=06f77ab650764027b3648c3f650f13ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.252707 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4134ba0b3d324d69b69201205a09725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.252768 20664 tablet_bootstrap.cc:492] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:33.252849 20664 ts_tablet_manager.cc:1367] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.001s
I20240502 20:45:33.253535 20664 raft_consensus.cc:348] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:33.253649 20664 raft_consensus.cc:374] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.253684 20664 raft_consensus.cc:729] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.253811 20664 consensus_queue.cc:260] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:33.253907 20656 tablet_bootstrap.cc:492] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:33.253993 20656 ts_tablet_manager.cc:1367] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 20:45:33.254098 20664 ts_tablet_manager.cc:1398] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:33.254287 20664 tablet_bootstrap.cc:492] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:33.254552 20656 raft_consensus.cc:348] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:33.254671 20656 raft_consensus.cc:374] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.254710 20656 raft_consensus.cc:729] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.254850 20656 consensus_queue.cc:260] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:33.255045 20659 tablet_bootstrap.cc:492] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:33.255110 20656 ts_tablet_manager.cc:1398] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:33.255209 20664 tablet_bootstrap.cc:654] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.255255 20656 tablet_bootstrap.cc:492] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:33.255476 20657 tablet_bootstrap.cc:654] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.255944 20659 tablet_bootstrap.cc:654] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.256155 20656 tablet_bootstrap.cc:654] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.256846 20664 tablet_bootstrap.cc:492] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:33.256914 20664 ts_tablet_manager.cc:1367] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:33.257681 20662 tablet_bootstrap.cc:492] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:33.257774 20662 ts_tablet_manager.cc:1367] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.000s
I20240502 20:45:33.258262 20664 raft_consensus.cc:348] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.258304 20662 raft_consensus.cc:348] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.258405 20664 raft_consensus.cc:374] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.258442 20662 raft_consensus.cc:374] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.258479 20662 raft_consensus.cc:729] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.258489 20664 raft_consensus.cc:729] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.258602 20657 tablet_bootstrap.cc:492] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:33.258625 20662 consensus_queue.cc:260] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.258702 20657 ts_tablet_manager.cc:1367] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.001s
I20240502 20:45:33.258711 20656 tablet_bootstrap.cc:492] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:33.258785 20656 ts_tablet_manager.cc:1367] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:33.258796 20664 consensus_queue.cc:260] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.258921 20662 ts_tablet_manager.cc:1398] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:33.259141 20664 ts_tablet_manager.cc:1398] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:33.259191 20657 raft_consensus.cc:348] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.259238 20656 raft_consensus.cc:348] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.259318 20657 raft_consensus.cc:374] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.259333 20656 raft_consensus.cc:374] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.259352 20657 raft_consensus.cc:729] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.259364 20656 raft_consensus.cc:729] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.259430 20659 tablet_bootstrap.cc:492] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:33.259505 20659 ts_tablet_manager.cc:1367] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:33.259469 20657 consensus_queue.cc:260] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.259480 20656 consensus_queue.cc:260] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.259732 20657 ts_tablet_manager.cc:1398] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:33.259744 20656 ts_tablet_manager.cc:1398] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:33.259977 20659 raft_consensus.cc:348] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.260097 20659 raft_consensus.cc:374] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.260135 20659 raft_consensus.cc:729] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.260251 20659 consensus_queue.cc:260] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.260675 20659 ts_tablet_manager.cc:1398] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:33.308153 20667 raft_consensus.cc:482] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:33.308301 20667 raft_consensus.cc:504] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:33.308748 20667 leader_election.cc:288] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:33.309324 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df575a25f570474f8b67748701025faa" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8" is_pre_election: true
I20240502 20:45:33.309374 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df575a25f570474f8b67748701025faa" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:33.309505 20585 raft_consensus.cc:2454] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:33.309505 20046 raft_consensus.cc:2454] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:33.309834 20249 leader_election.cc:302] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:33.310035 20667 raft_consensus.cc:2781] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:33.310097 20667 raft_consensus.cc:482] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:33.310139 20667 raft_consensus.cc:3037] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.311057 20667 raft_consensus.cc:504] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:33.311431 20667 leader_election.cc:288] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Requested vote from peers 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:33.311818 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df575a25f570474f8b67748701025faa" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8"
I20240502 20:45:33.311872 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df575a25f570474f8b67748701025faa" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:33.311959 20046 raft_consensus.cc:3037] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.311976 20585 raft_consensus.cc:3037] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.312762 20585 raft_consensus.cc:2454] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:33.312805 20046 raft_consensus.cc:2454] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:33.313058 20249 leader_election.cc:302] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:33.313261 20667 raft_consensus.cc:2781] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:33.313356 20667 raft_consensus.cc:686] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [term 1 LEADER]: Becoming Leader. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Running, Role: LEADER
I20240502 20:45:33.313515 20667 consensus_queue.cc:237] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:33.315371 19762 catalog_manager.cc:5484] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af reported cstate change: term changed from 0 to 1, leader changed from <none> to c302712f8dbd4fa7bed797758fd772af (127.19.66.132). New cstate: current_term: 1 leader_uuid: "c302712f8dbd4fa7bed797758fd772af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:33.354476 20672 raft_consensus.cc:482] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:33.354666 20672 raft_consensus.cc:504] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.356559 20672 leader_election.cc:288] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:33.361349 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4134ba0b3d324d69b69201205a09725d" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:33.361550 20585 raft_consensus.cc:2454] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:33.361971 19981 leader_election.cc:302] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:33.362071 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4134ba0b3d324d69b69201205a09725d" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:33.362304 20318 raft_consensus.cc:2454] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:33.362318 20672 raft_consensus.cc:2781] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:33.362383 20672 raft_consensus.cc:482] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:33.362442 20672 raft_consensus.cc:3037] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.364252 20672 raft_consensus.cc:504] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.364813 20672 leader_election.cc:288] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Requested vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:33.365392 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4134ba0b3d324d69b69201205a09725d" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:33.365392 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4134ba0b3d324d69b69201205a09725d" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:33.365528 20585 raft_consensus.cc:3037] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.365528 20318 raft_consensus.cc:3037] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.366549 20585 raft_consensus.cc:2454] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:33.366549 20318 raft_consensus.cc:2454] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:33.366983 19979 leader_election.cc:302] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:33.367203 20672 raft_consensus.cc:2781] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:33.367507 20672 raft_consensus.cc:686] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [term 1 LEADER]: Becoming Leader. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Running, Role: LEADER
I20240502 20:45:33.367902 20672 consensus_queue.cc:237] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.371299 19762 catalog_manager.cc:5484] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951291b2e23f46a79812e404a06690d8 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "951291b2e23f46a79812e404a06690d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:33.476217 20673 raft_consensus.cc:482] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:33.476390 20673 raft_consensus.cc:504] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.477509 20673 leader_election.cc:288] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:33.481892 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90fd3f27ca0d4e70afc9b2e680455a88" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:33.482139 20452 raft_consensus.cc:2454] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:33.482385 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90fd3f27ca0d4e70afc9b2e680455a88" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:33.482539 19911 raft_consensus.cc:2454] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:33.482584 20112 leader_election.cc:302] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 
I20240502 20:45:33.482852 20673 raft_consensus.cc:2781] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:33.482939 20673 raft_consensus.cc:482] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:33.482985 20673 raft_consensus.cc:3037] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.483883 20673 raft_consensus.cc:504] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.484212 20673 leader_election.cc:288] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Requested vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:33.484578 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90fd3f27ca0d4e70afc9b2e680455a88" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:33.484628 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90fd3f27ca0d4e70afc9b2e680455a88" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:33.484733 20452 raft_consensus.cc:3037] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.484742 19911 raft_consensus.cc:3037] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.485766 20452 raft_consensus.cc:2454] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:33.485764 19911 raft_consensus.cc:2454] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:33.486060 20112 leader_election.cc:302] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:33.486248 20673 raft_consensus.cc:2781] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:33.486464 20673 raft_consensus.cc:686] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 LEADER]: Becoming Leader. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Running, Role: LEADER
I20240502 20:45:33.486769 20673 consensus_queue.cc:237] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.488988 19762 catalog_manager.cc:5484] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131). New cstate: current_term: 1 leader_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:33.609458 20768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:33.609719 20768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:33.612987 20768 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 20:45:33.715976 20585 raft_consensus.cc:1261] T df575a25f570474f8b67748701025faa P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Refusing update from remote peer c302712f8dbd4fa7bed797758fd772af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.716969 20318 raft_consensus.cc:1261] T 4134ba0b3d324d69b69201205a09725d P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.717268 20046 raft_consensus.cc:1261] T df575a25f570474f8b67748701025faa P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Refusing update from remote peer c302712f8dbd4fa7bed797758fd772af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.719110 20667 consensus_queue.cc:1035] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.719813 20585 raft_consensus.cc:1261] T 4134ba0b3d324d69b69201205a09725d P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.720167 20672 consensus_queue.cc:1035] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.720968 20672 consensus_queue.cc:1035] T 4134ba0b3d324d69b69201205a09725d P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.723989 20679 consensus_queue.cc:1035] T df575a25f570474f8b67748701025faa P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.726238 20452 raft_consensus.cc:1261] T 90fd3f27ca0d4e70afc9b2e680455a88 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.729211 19911 raft_consensus.cc:1261] T 90fd3f27ca0d4e70afc9b2e680455a88 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:33.729897 20766 consensus_queue.cc:1035] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:33.730432 20766 consensus_queue.cc:1035] T 90fd3f27ca0d4e70afc9b2e680455a88 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 133.067 ms
  time per row: 0.0665334 ms
I20240502 20:45:33.800585 19760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:33.806912 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5aa53b956b4b129f4cd470f4102f57 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19fdc2a2a11c44bdab4ea9afa979b8bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.806912 20157 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5aa53b956b4b129f4cd470f4102f57 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19fdc2a2a11c44bdab4ea9afa979b8bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.807252 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f5aa53b956b4b129f4cd470f4102f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.807252 20157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f5aa53b956b4b129f4cd470f4102f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.807538 20026 tablet_service.cc:1467] Processing CreateTablet for tablet e81719d47a2c4d3e8d35ebc674d4fb2a (DEFAULT_TABLE table=rebalance_test_table_2 [id=19fdc2a2a11c44bdab4ea9afa979b8bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.807852 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e81719d47a2c4d3e8d35ebc674d4fb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.807833 20296 tablet_service.cc:1467] Processing CreateTablet for tablet e81719d47a2c4d3e8d35ebc674d4fb2a (DEFAULT_TABLE table=rebalance_test_table_2 [id=19fdc2a2a11c44bdab4ea9afa979b8bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.808091 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e81719d47a2c4d3e8d35ebc674d4fb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.810194 20836 tablet_bootstrap.cc:492] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:33.810254 20565 tablet_service.cc:1467] Processing CreateTablet for tablet e81719d47a2c4d3e8d35ebc674d4fb2a (DEFAULT_TABLE table=rebalance_test_table_2 [id=19fdc2a2a11c44bdab4ea9afa979b8bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.810551 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e81719d47a2c4d3e8d35ebc674d4fb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.811057 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 9341fb76068f419b8f2cb05e30cffa9f (DEFAULT_TABLE table=rebalance_test_table_2 [id=19fdc2a2a11c44bdab4ea9afa979b8bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.811075 19890 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5aa53b956b4b129f4cd470f4102f57 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19fdc2a2a11c44bdab4ea9afa979b8bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.811302 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9341fb76068f419b8f2cb05e30cffa9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.811374 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f5aa53b956b4b129f4cd470f4102f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.811507 20834 tablet_bootstrap.cc:492] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:33.812515 20834 tablet_bootstrap.cc:654] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.813298 20157 tablet_service.cc:1467] Processing CreateTablet for tablet 9341fb76068f419b8f2cb05e30cffa9f (DEFAULT_TABLE table=rebalance_test_table_2 [id=19fdc2a2a11c44bdab4ea9afa979b8bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.813663 20157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9341fb76068f419b8f2cb05e30cffa9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.813884 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 9341fb76068f419b8f2cb05e30cffa9f (DEFAULT_TABLE table=rebalance_test_table_2 [id=19fdc2a2a11c44bdab4ea9afa979b8bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:33.813923 20836 tablet_bootstrap.cc:654] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.814113 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9341fb76068f419b8f2cb05e30cffa9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:33.814635 20837 tablet_bootstrap.cc:492] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:33.815124 20836 tablet_bootstrap.cc:492] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:33.815222 20836 ts_tablet_manager.cc:1367] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:33.815743 20836 raft_consensus.cc:348] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.815902 20836 raft_consensus.cc:374] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.815913 20835 tablet_bootstrap.cc:492] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:33.815965 20836 raft_consensus.cc:729] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.816148 20836 consensus_queue.cc:260] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.816534 20836 ts_tablet_manager.cc:1398] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:33.816843 20835 tablet_bootstrap.cc:654] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.817221 20840 tablet_bootstrap.cc:492] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:33.817646 20837 tablet_bootstrap.cc:654] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.818770 20838 tablet_bootstrap.cc:492] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:33.819042 20837 tablet_bootstrap.cc:492] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:33.819042 20835 tablet_bootstrap.cc:492] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:33.819128 20835 ts_tablet_manager.cc:1367] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:33.819128 20837 ts_tablet_manager.cc:1367] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:33.819561 20835 raft_consensus.cc:348] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.819556 20837 raft_consensus.cc:348] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.819693 20837 raft_consensus.cc:374] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.819693 20835 raft_consensus.cc:374] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.819728 20837 raft_consensus.cc:729] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.819728 20835 raft_consensus.cc:729] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.819751 20838 tablet_bootstrap.cc:654] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.819856 20835 consensus_queue.cc:260] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.819856 20837 consensus_queue.cc:260] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.820117 20837 ts_tablet_manager.cc:1398] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:33.820117 20835 ts_tablet_manager.cc:1398] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:33.820281 20835 tablet_bootstrap.cc:492] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:33.820847 20840 tablet_bootstrap.cc:654] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.821132 20834 tablet_bootstrap.cc:492] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:33.821192 20835 tablet_bootstrap.cc:654] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.821357 20834 ts_tablet_manager.cc:1367] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 20:45:33.821907 20834 raft_consensus.cc:348] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.822036 20834 raft_consensus.cc:374] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.822072 20834 raft_consensus.cc:729] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.822701 20835 tablet_bootstrap.cc:492] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:33.822818 20835 ts_tablet_manager.cc:1367] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:45:33.822945 20840 tablet_bootstrap.cc:492] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:33.822877 20834 consensus_queue.cc:260] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.823031 20840 ts_tablet_manager.cc:1367] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.002s
I20240502 20:45:33.823287 20834 ts_tablet_manager.cc:1398] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:33.823474 20834 tablet_bootstrap.cc:492] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:33.824502 20834 tablet_bootstrap.cc:654] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.824205 20835 raft_consensus.cc:348] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.824558 20835 raft_consensus.cc:374] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.824647 20835 raft_consensus.cc:729] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.824872 20835 consensus_queue.cc:260] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.825819 20834 tablet_bootstrap.cc:492] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:33.825917 20834 ts_tablet_manager.cc:1367] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:33.825981 20835 ts_tablet_manager.cc:1398] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240502 20:45:33.826431 20834 raft_consensus.cc:348] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.826521 20834 raft_consensus.cc:374] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.826545 20834 raft_consensus.cc:729] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.826663 20834 consensus_queue.cc:260] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.826946 20834 ts_tablet_manager.cc:1398] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:33.826956 20840 raft_consensus.cc:348] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.827296 20840 raft_consensus.cc:374] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.827359 20840 raft_consensus.cc:729] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.827505 20840 consensus_queue.cc:260] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.827814 20840 ts_tablet_manager.cc:1398] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.001s
I20240502 20:45:33.828013 20840 tablet_bootstrap.cc:492] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:33.829157 20840 tablet_bootstrap.cc:654] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:33.830458 20838 tablet_bootstrap.cc:492] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:33.830585 20838 ts_tablet_manager.cc:1367] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.000s
I20240502 20:45:33.831051 20838 raft_consensus.cc:348] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.831179 20838 raft_consensus.cc:374] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.831212 20838 raft_consensus.cc:729] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.831219 20840 tablet_bootstrap.cc:492] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:33.831313 20840 ts_tablet_manager.cc:1367] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:45:33.831364 20838 consensus_queue.cc:260] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.831743 20838 ts_tablet_manager.cc:1398] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:33.831822 20840 raft_consensus.cc:348] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.831948 20840 raft_consensus.cc:374] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:33.831986 20840 raft_consensus.cc:729] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:33.832142 20840 consensus_queue.cc:260] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.832437 20840 ts_tablet_manager.cc:1398] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:45:33.847648 20674 raft_consensus.cc:482] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:33.847836 20674 raft_consensus.cc:504] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.848997 20674 leader_election.cc:288] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:33.853158 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9341fb76068f419b8f2cb05e30cffa9f" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:33.853331 20452 raft_consensus.cc:2454] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:33.853708 19843 leader_election.cc:302] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:33.853724 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9341fb76068f419b8f2cb05e30cffa9f" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" is_pre_election: true
I20240502 20:45:33.853854 20179 raft_consensus.cc:2454] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:33.853964 20674 raft_consensus.cc:2781] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:33.854022 20674 raft_consensus.cc:482] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:33.854068 20674 raft_consensus.cc:3037] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.854985 20674 raft_consensus.cc:504] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.855348 20674 leader_election.cc:288] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Requested vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:33.855648 20672 raft_consensus.cc:482] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:33.855885 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9341fb76068f419b8f2cb05e30cffa9f" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:33.855851 20672 raft_consensus.cc:504] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.856019 20452 raft_consensus.cc:3037] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.856110 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9341fb76068f419b8f2cb05e30cffa9f" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
I20240502 20:45:33.856272 20179 raft_consensus.cc:3037] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.856493 20672 leader_election.cc:288] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:33.856951 20452 raft_consensus.cc:2454] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:33.857617 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81719d47a2c4d3e8d35ebc674d4fb2a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:33.857633 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81719d47a2c4d3e8d35ebc674d4fb2a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:33.857831 20585 raft_consensus.cc:2454] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:33.857834 20318 raft_consensus.cc:2454] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:33.858273 19843 leader_election.cc:302] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:33.858444 19981 leader_election.cc:302] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:33.858683 20674 raft_consensus.cc:2781] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:33.858773 20672 raft_consensus.cc:2781] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:33.858832 20672 raft_consensus.cc:482] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:33.858867 20672 raft_consensus.cc:3037] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.859009 20674 raft_consensus.cc:686] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [term 1 LEADER]: Becoming Leader. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Running, Role: LEADER
I20240502 20:45:33.859479 20179 raft_consensus.cc:2454] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:33.859508 20674 consensus_queue.cc:237] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:33.859997 20672 raft_consensus.cc:504] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.860453 20672 leader_election.cc:288] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Requested vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:33.861018 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81719d47a2c4d3e8d35ebc674d4fb2a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:33.861259 20318 raft_consensus.cc:3037] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.862337 20318 raft_consensus.cc:2454] T e81719d47a2c4d3e8d35ebc674d4fb2a P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:33.862805 19979 leader_election.cc:302] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:33.862836 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81719d47a2c4d3e8d35ebc674d4fb2a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:33.863013 20585 raft_consensus.cc:3037] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:33.863024 20672 raft_consensus.cc:2781] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:33.862879 19761 catalog_manager.cc:5484] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91942b13253244aab62c2a52e1fdd827 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "91942b13253244aab62c2a52e1fdd827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:33.863127 20672 raft_consensus.cc:686] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [term 1 LEADER]: Becoming Leader. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Running, Role: LEADER
I20240502 20:45:33.863273 20672 consensus_queue.cc:237] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:33.864041 20585 raft_consensus.cc:2454] T e81719d47a2c4d3e8d35ebc674d4fb2a P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:33.866931 19761 catalog_manager.cc:5484] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951291b2e23f46a79812e404a06690d8 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "951291b2e23f46a79812e404a06690d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:34.057992 20829 raft_consensus.cc:482] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:34.058166 20829 raft_consensus.cc:504] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:34.058853 20829 leader_election.cc:288] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:34.059574 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5aa53b956b4b129f4cd470f4102f57" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:34.059599 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5aa53b956b4b129f4cd470f4102f57" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:34.059823 19911 raft_consensus.cc:2454] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:34.059823 20452 raft_consensus.cc:2454] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:34.060401 20112 leader_election.cc:302] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:34.060633 20829 raft_consensus.cc:2781] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:34.060709 20829 raft_consensus.cc:482] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:34.060760 20829 raft_consensus.cc:3037] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.061714 20829 raft_consensus.cc:504] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:34.062068 20829 leader_election.cc:288] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Requested vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:34.062589 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5aa53b956b4b129f4cd470f4102f57" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:34.062593 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5aa53b956b4b129f4cd470f4102f57" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:34.062708 20452 raft_consensus.cc:3037] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.062721 19911 raft_consensus.cc:3037] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.063495 20452 raft_consensus.cc:2454] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:34.063552 19911 raft_consensus.cc:2454] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:34.063818 20112 leader_election.cc:302] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 
I20240502 20:45:34.064067 20829 raft_consensus.cc:2781] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:34.064180 20829 raft_consensus.cc:686] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 LEADER]: Becoming Leader. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Running, Role: LEADER
I20240502 20:45:34.064347 20829 consensus_queue.cc:237] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:34.066219 19761 catalog_manager.cc:5484] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131). New cstate: current_term: 1 leader_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:34.235430 20855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:34.235862 20855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:34.239151 20855 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 20:45:34.265297 20672 consensus_queue.cc:1035] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:34.269381 20672 consensus_queue.cc:1035] T e81719d47a2c4d3e8d35ebc674d4fb2a P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:34.339623 19911 raft_consensus.cc:1261] T 8f5aa53b956b4b129f4cd470f4102f57 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:34.340119 20829 consensus_queue.cc:1035] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:34.344414 20452 raft_consensus.cc:1261] T 8f5aa53b956b4b129f4cd470f4102f57 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:34.344951 20829 consensus_queue.cc:1035] T 8f5aa53b956b4b129f4cd470f4102f57 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:34.347569 20451 raft_consensus.cc:1261] T 9341fb76068f419b8f2cb05e30cffa9f P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:34.348013 20179 raft_consensus.cc:1261] T 9341fb76068f419b8f2cb05e30cffa9f P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:34.348105 20674 consensus_queue.cc:1035] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:34.348711 20674 consensus_queue.cc:1035] T 9341fb76068f419b8f2cb05e30cffa9f P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 112.36 ms
  time per row: 0.0561802 ms
I20240502 20:45:34.402670 19760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:34.408877 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1c39c6e15949b3b2fd13a788769a7a (DEFAULT_TABLE table=rebalance_test_table_3 [id=64afcf4c55f04397b430c654e34f62e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:34.409245 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1c39c6e15949b3b2fd13a788769a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.409556 20157 tablet_service.cc:1467] Processing CreateTablet for tablet 9a81c4b4519d4be5a1be4d264fc29c74 (DEFAULT_TABLE table=rebalance_test_table_3 [id=64afcf4c55f04397b430c654e34f62e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:34.409842 20157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a81c4b4519d4be5a1be4d264fc29c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.409873 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 9a81c4b4519d4be5a1be4d264fc29c74 (DEFAULT_TABLE table=rebalance_test_table_3 [id=64afcf4c55f04397b430c654e34f62e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:34.410151 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a81c4b4519d4be5a1be4d264fc29c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.412546 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 5a32010c8c8b42c0a464e4d9f4fd849b (DEFAULT_TABLE table=rebalance_test_table_3 [id=64afcf4c55f04397b430c654e34f62e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:34.412547 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 9a81c4b4519d4be5a1be4d264fc29c74 (DEFAULT_TABLE table=rebalance_test_table_3 [id=64afcf4c55f04397b430c654e34f62e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:34.412856 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a81c4b4519d4be5a1be4d264fc29c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.412909 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a32010c8c8b42c0a464e4d9f4fd849b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.413513 20565 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1c39c6e15949b3b2fd13a788769a7a (DEFAULT_TABLE table=rebalance_test_table_3 [id=64afcf4c55f04397b430c654e34f62e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:34.413767 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1c39c6e15949b3b2fd13a788769a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.415699 20924 tablet_bootstrap.cc:492] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:34.415994 20564 tablet_service.cc:1467] Processing CreateTablet for tablet 5a32010c8c8b42c0a464e4d9f4fd849b (DEFAULT_TABLE table=rebalance_test_table_3 [id=64afcf4c55f04397b430c654e34f62e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:34.416224 20564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a32010c8c8b42c0a464e4d9f4fd849b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.416724 20924 tablet_bootstrap.cc:654] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:34.417752 20025 tablet_service.cc:1467] Processing CreateTablet for tablet 5a32010c8c8b42c0a464e4d9f4fd849b (DEFAULT_TABLE table=rebalance_test_table_3 [id=64afcf4c55f04397b430c654e34f62e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:34.418036 20025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a32010c8c8b42c0a464e4d9f4fd849b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.418211 20924 tablet_bootstrap.cc:492] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:34.418288 20924 ts_tablet_manager.cc:1367] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:34.418781 20924 raft_consensus.cc:348] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:34.418886 20924 raft_consensus.cc:374] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:34.418913 20924 raft_consensus.cc:729] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:34.419036 20924 consensus_queue.cc:260] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:34.419292 20924 ts_tablet_manager.cc:1398] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:34.417727 20026 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1c39c6e15949b3b2fd13a788769a7a (DEFAULT_TABLE table=rebalance_test_table_3 [id=64afcf4c55f04397b430c654e34f62e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:34.420861 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1c39c6e15949b3b2fd13a788769a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:34.422227 20922 tablet_bootstrap.cc:492] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:34.423095 20922 tablet_bootstrap.cc:654] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:34.423859 20925 tablet_bootstrap.cc:492] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:34.424165 20923 tablet_bootstrap.cc:492] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:34.424336 20922 tablet_bootstrap.cc:492] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:34.424463 20922 ts_tablet_manager.cc:1367] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:34.425020 20922 raft_consensus.cc:348] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:34.425297 20922 raft_consensus.cc:374] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:34.425487 20925 tablet_bootstrap.cc:654] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:34.425540 20922 raft_consensus.cc:729] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:34.425714 20927 tablet_bootstrap.cc:492] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:34.425768 20922 consensus_queue.cc:260] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:34.426131 20922 ts_tablet_manager.cc:1398] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:34.426630 20923 tablet_bootstrap.cc:654] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:34.426877 20927 tablet_bootstrap.cc:654] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:34.427807 20921 tablet_bootstrap.cc:492] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:34.427963 20918 raft_consensus.cc:482] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:34.428084 20918 raft_consensus.cc:504] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:34.428552 20918 leader_election.cc:288] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:34.428958 20921 tablet_bootstrap.cc:654] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:34.429129 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a81c4b4519d4be5a1be4d264fc29c74" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:34.429311 19911 raft_consensus.cc:2454] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:34.429322 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a81c4b4519d4be5a1be4d264fc29c74" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:34.429562 20923 tablet_bootstrap.cc:492] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:34.429736 20923 ts_tablet_manager.cc:1367] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.001s
W20240502 20:45:34.430127 20112 leader_election.cc:341] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211): Illegal state: must be running to vote when last-logged opid is not known
I20240502 20:45:34.430259 20923 raft_consensus.cc:348] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:34.430327 20112 leader_election.cc:302] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 5742a696d4c3406086b710e571e2cf0e
I20240502 20:45:34.430387 20927 tablet_bootstrap.cc:492] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:34.430395 20923 raft_consensus.cc:374] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:34.430473 20927 ts_tablet_manager.cc:1367] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:34.430465 20923 raft_consensus.cc:729] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:34.430617 20918 raft_consensus.cc:2781] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:34.430685 20918 raft_consensus.cc:482] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:34.430740 20918 raft_consensus.cc:3037] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.430728 20923 consensus_queue.cc:260] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:34.431028 20927 raft_consensus.cc:348] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:34.431152 20927 raft_consensus.cc:374] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:34.431151 20923 ts_tablet_manager.cc:1398] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:45:34.431200 20927 raft_consensus.cc:729] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:34.431388 20927 consensus_queue.cc:260] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:34.431844 20927 ts_tablet_manager.cc:1398] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:34.431962 20925 tablet_bootstrap.cc:492] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:34.431931 20918 raft_consensus.cc:504] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:34.432065 20925 ts_tablet_manager.cc:1367] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 20:45:34.432031 20927 tablet_bootstrap.cc:492] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:34.432324 20918 leader_election.cc:288] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Requested vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:34.432587 20925 raft_consensus.cc:348] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:34.432693 20925 raft_consensus.cc:374] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:34.432730 20925 raft_consensus.cc:729] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:34.432883 20925 consensus_queue.cc:260] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:34.433208 20925 ts_tablet_manager.cc:1398] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:34.433346 20927 tablet_bootstrap.cc:654] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:34.433389 20925 tablet_bootstrap.cc:492] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:34.434412 20925 tablet_bootstrap.cc:654] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:34.434743 20927 tablet_bootstrap.cc:492] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:34.434839 20927 ts_tablet_manager.cc:1367] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:45:34.435696 20925 tablet_bootstrap.cc:492] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:34.435766 20921 tablet_bootstrap.cc:492] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:34.435796 20925 ts_tablet_manager.cc:1367] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:34.435887 20921 ts_tablet_manager.cc:1367] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 20:45:34.436218 20925 raft_consensus.cc:348] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:34.436347 20925 raft_consensus.cc:374] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:34.436379 20925 raft_consensus.cc:729] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:34.436371 20921 raft_consensus.cc:348] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:34.436494 20921 raft_consensus.cc:374] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:34.436532 20921 raft_consensus.cc:729] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:34.436511 20925 consensus_queue.cc:260] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:34.436781 20925 ts_tablet_manager.cc:1398] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:34.436686 20921 consensus_queue.cc:260] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:34.437160 20921 ts_tablet_manager.cc:1398] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:34.437511 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a81c4b4519d4be5a1be4d264fc29c74" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:34.437608 20921 tablet_bootstrap.cc:492] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:34.437580 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a81c4b4519d4be5a1be4d264fc29c74" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:34.437652 19911 raft_consensus.cc:3037] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.437723 20452 raft_consensus.cc:3037] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.438675 20921 tablet_bootstrap.cc:654] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:34.438701 20452 raft_consensus.cc:2454] T 9a81c4b4519d4be5a1be4d264fc29c74 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:34.438701 19911 raft_consensus.cc:2454] T 9a81c4b4519d4be5a1be4d264fc29c74 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:34.439180 20112 leader_election.cc:302] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:34.439433 20918 raft_consensus.cc:2781] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:34.439520 20918 raft_consensus.cc:686] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 LEADER]: Becoming Leader. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Running, Role: LEADER
I20240502 20:45:34.439581 20927 raft_consensus.cc:348] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:34.439698 20927 raft_consensus.cc:374] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:34.439730 20927 raft_consensus.cc:729] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:34.439698 20918 consensus_queue.cc:237] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:34.439908 20927 consensus_queue.cc:260] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:34.440445 20927 ts_tablet_manager.cc:1398] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.001s
I20240502 20:45:34.443104 19765 catalog_manager.cc:5484] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131). New cstate: current_term: 1 leader_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:34.445174 20921 tablet_bootstrap.cc:492] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:34.445297 20921 ts_tablet_manager.cc:1367] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 20:45:34.446066 20921 raft_consensus.cc:348] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:34.446170 20921 raft_consensus.cc:374] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:34.446206 20921 raft_consensus.cc:729] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:34.446347 20921 consensus_queue.cc:260] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:34.446595 20921 ts_tablet_manager.cc:1398] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:34.489455 20900 raft_consensus.cc:482] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:34.489620 20900 raft_consensus.cc:504] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:34.490123 20900 leader_election.cc:288] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:34.490604 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1c39c6e15949b3b2fd13a788769a7a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:34.490661 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1c39c6e15949b3b2fd13a788769a7a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:34.490751 20318 raft_consensus.cc:2454] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:34.490836 20585 raft_consensus.cc:2454] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:34.491058 19979 leader_election.cc:302] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:34.491276 20900 raft_consensus.cc:2781] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:34.491338 20900 raft_consensus.cc:482] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:34.491366 20900 raft_consensus.cc:3037] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.492262 20900 raft_consensus.cc:504] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:34.492611 20900 leader_election.cc:288] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Requested vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:34.493256 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1c39c6e15949b3b2fd13a788769a7a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:34.493256 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1c39c6e15949b3b2fd13a788769a7a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:34.493579 20585 raft_consensus.cc:3037] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.493575 20318 raft_consensus.cc:3037] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.494658 20318 raft_consensus.cc:2454] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:34.494658 20585 raft_consensus.cc:2454] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:34.495019 19981 leader_election.cc:302] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:34.495245 20900 raft_consensus.cc:2781] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:34.495347 20900 raft_consensus.cc:686] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [term 1 LEADER]: Becoming Leader. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Running, Role: LEADER
I20240502 20:45:34.495519 20900 consensus_queue.cc:237] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:34.497484 19765 catalog_manager.cc:5484] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951291b2e23f46a79812e404a06690d8 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "951291b2e23f46a79812e404a06690d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:34.698622 20667 raft_consensus.cc:482] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:34.698784 20667 raft_consensus.cc:504] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:34.699544 20667 leader_election.cc:288] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:34.700026 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a32010c8c8b42c0a464e4d9f4fd849b" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8" is_pre_election: true
I20240502 20:45:34.700039 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a32010c8c8b42c0a464e4d9f4fd849b" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:34.700197 20585 raft_consensus.cc:2454] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:34.700197 20045 raft_consensus.cc:2454] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:34.700526 20249 leader_election.cc:302] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:34.700696 20667 raft_consensus.cc:2781] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:34.700757 20667 raft_consensus.cc:482] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:34.700798 20667 raft_consensus.cc:3037] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.702075 20667 raft_consensus.cc:504] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:34.702585 20667 leader_election.cc:288] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Requested vote from peers 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:34.703030 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a32010c8c8b42c0a464e4d9f4fd849b" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8"
I20240502 20:45:34.703049 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a32010c8c8b42c0a464e4d9f4fd849b" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:34.703183 20045 raft_consensus.cc:3037] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.703183 20585 raft_consensus.cc:3037] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:34.704265 20585 raft_consensus.cc:2454] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:34.704265 20045 raft_consensus.cc:2454] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:34.704586 20249 leader_election.cc:302] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:34.704808 20667 raft_consensus.cc:2781] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:34.705011 20667 raft_consensus.cc:686] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [term 1 LEADER]: Becoming Leader. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Running, Role: LEADER
I20240502 20:45:34.705176 20667 consensus_queue.cc:237] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:34.706904 19765 catalog_manager.cc:5484] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af reported cstate change: term changed from 0 to 1, leader changed from <none> to c302712f8dbd4fa7bed797758fd772af (127.19.66.132). New cstate: current_term: 1 leader_uuid: "c302712f8dbd4fa7bed797758fd772af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:34.852541 20940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:34.852833 20940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:34.856048 20940 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:45:34.860898 20829 consensus_queue.cc:1035] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:34.864419 20766 consensus_queue.cc:1035] T 9a81c4b4519d4be5a1be4d264fc29c74 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 20:45:34.960453 20585 raft_consensus.cc:1261] T 5b1c39c6e15949b3b2fd13a788769a7a P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:34.960950 20318 raft_consensus.cc:1261] T 5b1c39c6e15949b3b2fd13a788769a7a P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:34.961053 20900 consensus_queue.cc:1035] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:34.961937 20900 consensus_queue.cc:1035] T 5b1c39c6e15949b3b2fd13a788769a7a P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:34.976157 20585 raft_consensus.cc:1261] T 5a32010c8c8b42c0a464e4d9f4fd849b P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Refusing update from remote peer c302712f8dbd4fa7bed797758fd772af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:34.976696 20667 consensus_queue.cc:1035] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:34.983157 20045 raft_consensus.cc:1261] T 5a32010c8c8b42c0a464e4d9f4fd849b P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Refusing update from remote peer c302712f8dbd4fa7bed797758fd772af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:34.983698 20667 consensus_queue.cc:1035] T 5a32010c8c8b42c0a464e4d9f4fd849b P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 119.652 ms
  time per row: 0.0598261 ms
I20240502 20:45:35.030414 19764 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:35.038045 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 41166a0fb05d4371b6c0c064d169ada4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f060242baf994315a1ce1cedc129b2eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.038431 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41166a0fb05d4371b6c0c064d169ada4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.038988 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 41166a0fb05d4371b6c0c064d169ada4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f060242baf994315a1ce1cedc129b2eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.039108 20157 tablet_service.cc:1467] Processing CreateTablet for tablet 41166a0fb05d4371b6c0c064d169ada4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f060242baf994315a1ce1cedc129b2eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.039273 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41166a0fb05d4371b6c0c064d169ada4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.039357 20157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41166a0fb05d4371b6c0c064d169ada4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.039716 20155 tablet_service.cc:1467] Processing CreateTablet for tablet 5a20e3845bcc45b19b0533da1efe1646 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f060242baf994315a1ce1cedc129b2eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.038414 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 20c70adc4eae4691b41a064fe8d87b68 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f060242baf994315a1ce1cedc129b2eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.040481 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a20e3845bcc45b19b0533da1efe1646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.041775 21004 tablet_bootstrap.cc:492] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:35.041920 20564 tablet_service.cc:1467] Processing CreateTablet for tablet 20c70adc4eae4691b41a064fe8d87b68 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f060242baf994315a1ce1cedc129b2eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.042181 20564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c70adc4eae4691b41a064fe8d87b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.042482 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c70adc4eae4691b41a064fe8d87b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.042690 21004 tablet_bootstrap.cc:654] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.044224 21004 tablet_bootstrap.cc:492] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:35.044310 21004 ts_tablet_manager.cc:1367] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:35.044808 21004 raft_consensus.cc:348] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.044824 20026 tablet_service.cc:1467] Processing CreateTablet for tablet 20c70adc4eae4691b41a064fe8d87b68 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f060242baf994315a1ce1cedc129b2eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.044927 21004 raft_consensus.cc:374] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.044957 21004 raft_consensus.cc:729] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.044973 21006 tablet_bootstrap.cc:492] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:35.045243 21004 consensus_queue.cc:260] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.045585 21004 ts_tablet_manager.cc:1398] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:35.045868 21004 tablet_bootstrap.cc:492] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:35.046494 21006 tablet_bootstrap.cc:654] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.046824 21004 tablet_bootstrap.cc:654] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.046787 20432 tablet_service.cc:1467] Processing CreateTablet for tablet 5a20e3845bcc45b19b0533da1efe1646 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f060242baf994315a1ce1cedc129b2eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.047042 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c70adc4eae4691b41a064fe8d87b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.047497 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 5a20e3845bcc45b19b0533da1efe1646 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f060242baf994315a1ce1cedc129b2eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.047812 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a20e3845bcc45b19b0533da1efe1646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.049499 21010 tablet_bootstrap.cc:492] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:35.050284 21004 tablet_bootstrap.cc:492] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:35.050442 21004 ts_tablet_manager.cc:1367] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:35.050789 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a20e3845bcc45b19b0533da1efe1646. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.050931 21004 raft_consensus.cc:348] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:35.051056 21004 raft_consensus.cc:374] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.051090 21004 raft_consensus.cc:729] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.051242 21004 consensus_queue.cc:260] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:35.051657 21004 ts_tablet_manager.cc:1398] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:35.052215 21011 tablet_bootstrap.cc:492] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:35.052762 21010 tablet_bootstrap.cc:654] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.053050 21003 tablet_bootstrap.cc:492] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:35.053369 21007 tablet_bootstrap.cc:492] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:35.054034 21003 tablet_bootstrap.cc:654] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.054541 21007 tablet_bootstrap.cc:654] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.054611 21011 tablet_bootstrap.cc:654] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.055454 21006 tablet_bootstrap.cc:492] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:35.055562 21006 ts_tablet_manager.cc:1367] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.000s
I20240502 20:45:35.055639 21011 tablet_bootstrap.cc:492] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:35.055718 21011 ts_tablet_manager.cc:1367] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:35.055826 21003 tablet_bootstrap.cc:492] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:35.055826 21010 tablet_bootstrap.cc:492] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:35.055912 21010 ts_tablet_manager.cc:1367] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 20:45:35.055912 21003 ts_tablet_manager.cc:1367] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:45:35.056070 21006 raft_consensus.cc:348] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:35.056113 21011 raft_consensus.cc:348] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.056185 21006 raft_consensus.cc:374] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.056198 21011 raft_consensus.cc:374] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.056221 21006 raft_consensus.cc:729] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.056232 21011 raft_consensus.cc:729] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.056370 21011 consensus_queue.cc:260] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.056386 21006 consensus_queue.cc:260] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:35.056403 21010 raft_consensus.cc:348] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:35.056519 21010 raft_consensus.cc:374] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.056563 21010 raft_consensus.cc:729] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.056509 21003 raft_consensus.cc:348] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.056613 21003 raft_consensus.cc:374] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.056641 21011 ts_tablet_manager.cc:1398] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:35.056648 21003 raft_consensus.cc:729] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.056696 21010 consensus_queue.cc:260] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:35.056807 21011 tablet_bootstrap.cc:492] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:35.056768 21003 consensus_queue.cc:260] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.057037 21003 ts_tablet_manager.cc:1398] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:45:35.057264 21003 tablet_bootstrap.cc:492] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:35.057492 21010 ts_tablet_manager.cc:1398] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:35.058095 21011 tablet_bootstrap.cc:654] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.058244 21003 tablet_bootstrap.cc:654] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.058394 21006 ts_tablet_manager.cc:1398] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:45:35.060184 21011 tablet_bootstrap.cc:492] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:35.060277 21011 ts_tablet_manager.cc:1367] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:35.060791 21011 raft_consensus.cc:348] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:35.061173 21011 raft_consensus.cc:374] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.061318 21011 raft_consensus.cc:729] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.061618 21011 consensus_queue.cc:260] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:35.062076 21011 ts_tablet_manager.cc:1398] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 20:45:35.063333 21003 tablet_bootstrap.cc:492] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:35.063437 21007 tablet_bootstrap.cc:492] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:35.063567 21007 ts_tablet_manager.cc:1367] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 20:45:35.063560 21003 ts_tablet_manager.cc:1367] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:45:35.064416 21003 raft_consensus.cc:348] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:35.064599 21003 raft_consensus.cc:374] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.064646 21003 raft_consensus.cc:729] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.064237 21007 raft_consensus.cc:348] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:35.064857 21003 consensus_queue.cc:260] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:35.064913 21007 raft_consensus.cc:374] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.064975 21007 raft_consensus.cc:729] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.065198 21007 consensus_queue.cc:260] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:35.065833 21003 ts_tablet_manager.cc:1398] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 20:45:35.065914 21007 ts_tablet_manager.cc:1398] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:35.086494 20916 raft_consensus.cc:482] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:35.086648 20916 raft_consensus.cc:504] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.087141 20916 leader_election.cc:288] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:35.087572 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41166a0fb05d4371b6c0c064d169ada4" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:35.087788 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41166a0fb05d4371b6c0c064d169ada4" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" is_pre_election: true
I20240502 20:45:35.087903 20452 raft_consensus.cc:2454] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:35.088069 20179 raft_consensus.cc:2454] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:35.088541 19843 leader_election.cc:302] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:35.088846 20916 raft_consensus.cc:2781] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:35.088935 20916 raft_consensus.cc:482] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:35.088985 20916 raft_consensus.cc:3037] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.090126 20916 raft_consensus.cc:504] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.090502 20916 leader_election.cc:288] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Requested vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:35.090972 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41166a0fb05d4371b6c0c064d169ada4" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
I20240502 20:45:35.090993 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41166a0fb05d4371b6c0c064d169ada4" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:35.091121 20179 raft_consensus.cc:3037] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.091130 20452 raft_consensus.cc:3037] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.092180 20452 raft_consensus.cc:2454] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:35.092644 19843 leader_election.cc:302] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:35.092703 20179 raft_consensus.cc:2454] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:35.092936 20916 raft_consensus.cc:2781] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:35.093225 20916 raft_consensus.cc:686] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [term 1 LEADER]: Becoming Leader. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Running, Role: LEADER
I20240502 20:45:35.093437 20916 consensus_queue.cc:237] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.095301 19763 catalog_manager.cc:5484] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91942b13253244aab62c2a52e1fdd827 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "91942b13253244aab62c2a52e1fdd827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:35.113616 20766 raft_consensus.cc:482] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:35.113770 20766 raft_consensus.cc:504] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:35.114208 20766 leader_election.cc:288] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:35.114760 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a20e3845bcc45b19b0533da1efe1646" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:35.114759 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a20e3845bcc45b19b0533da1efe1646" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:35.114920 19911 raft_consensus.cc:2454] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:35.114920 20452 raft_consensus.cc:2454] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:35.115227 20112 leader_election.cc:302] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:35.115409 20766 raft_consensus.cc:2781] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:35.115470 20766 raft_consensus.cc:482] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:35.115520 20766 raft_consensus.cc:3037] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.116384 20766 raft_consensus.cc:504] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:35.116701 20766 leader_election.cc:288] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Requested vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:35.117102 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a20e3845bcc45b19b0533da1efe1646" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:35.117187 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a20e3845bcc45b19b0533da1efe1646" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:35.117231 19911 raft_consensus.cc:3037] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.117311 20452 raft_consensus.cc:3037] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.118237 20452 raft_consensus.cc:2454] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:35.118237 19911 raft_consensus.cc:2454] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:35.118562 20112 leader_election.cc:302] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:35.118749 20766 raft_consensus.cc:2781] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:35.118834 20766 raft_consensus.cc:686] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 LEADER]: Becoming Leader. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Running, Role: LEADER
I20240502 20:45:35.118981 20766 consensus_queue.cc:237] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:35.120790 19763 catalog_manager.cc:5484] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131). New cstate: current_term: 1 leader_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:35.123948 20980 raft_consensus.cc:482] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:35.124056 20980 raft_consensus.cc:504] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:35.125155 20980 leader_election.cc:288] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:35.129688 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c70adc4eae4691b41a064fe8d87b68" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8" is_pre_election: true
I20240502 20:45:35.129735 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c70adc4eae4691b41a064fe8d87b68" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:35.129851 20045 raft_consensus.cc:2454] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 0.
I20240502 20:45:35.129909 20318 raft_consensus.cc:2454] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 0.
I20240502 20:45:35.130259 20520 leader_election.cc:302] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:35.130529 20980 raft_consensus.cc:2781] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:35.130591 20980 raft_consensus.cc:482] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:35.130621 20980 raft_consensus.cc:3037] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.131502 20980 raft_consensus.cc:504] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:35.131850 20980 leader_election.cc:288] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 election: Requested vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:35.132241 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c70adc4eae4691b41a064fe8d87b68" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:35.132390 20318 raft_consensus.cc:3037] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.132405 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c70adc4eae4691b41a064fe8d87b68" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8"
I20240502 20:45:35.132545 20045 raft_consensus.cc:3037] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.133224 20318 raft_consensus.cc:2454] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 1.
I20240502 20:45:35.133435 20045 raft_consensus.cc:2454] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 1.
I20240502 20:45:35.133556 20518 leader_election.cc:302] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:35.133740 20980 raft_consensus.cc:2781] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:35.133946 20980 raft_consensus.cc:686] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [term 1 LEADER]: Becoming Leader. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Running, Role: LEADER
I20240502 20:45:35.134282 20980 consensus_queue.cc:237] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:35.136693 19763 catalog_manager.cc:5484] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa7ea13bcb524353a03145dda1461c9e (127.19.66.134). New cstate: current_term: 1 leader_uuid: "fa7ea13bcb524353a03145dda1461c9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:35.270498 21026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:35.270768 21026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:35.274096 21026 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 20:45:35.361438 19911 raft_consensus.cc:1261] T 5a20e3845bcc45b19b0533da1efe1646 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:35.361657 20452 raft_consensus.cc:1261] T 5a20e3845bcc45b19b0533da1efe1646 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:35.362222 20829 consensus_queue.cc:1035] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:35.362670 20829 consensus_queue.cc:1035] T 5a20e3845bcc45b19b0533da1efe1646 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:35.364502 20045 raft_consensus.cc:1261] T 20c70adc4eae4691b41a064fe8d87b68 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Refusing update from remote peer fa7ea13bcb524353a03145dda1461c9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:35.365726 20980 consensus_queue.cc:1035] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:35.371398 20316 raft_consensus.cc:1261] T 20c70adc4eae4691b41a064fe8d87b68 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Refusing update from remote peer fa7ea13bcb524353a03145dda1461c9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:35.371960 20980 consensus_queue.cc:1035] T 20c70adc4eae4691b41a064fe8d87b68 P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:35.385785 20451 raft_consensus.cc:1261] T 41166a0fb05d4371b6c0c064d169ada4 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:35.386097 20179 raft_consensus.cc:1261] T 41166a0fb05d4371b6c0c064d169ada4 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:35.386807 20916 consensus_queue.cc:1035] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:35.394874 21018 consensus_queue.cc:1035] T 41166a0fb05d4371b6c0c064d169ada4 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 131.891 ms
  time per row: 0.0659455 ms
I20240502 20:45:35.454767 19765 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:35.461822 20026 tablet_service.cc:1467] Processing CreateTablet for tablet d77ddb854c034174babe1a753259cfc8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=134a4b9c14424ae188d91b77f058d927]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.462056 20025 tablet_service.cc:1467] Processing CreateTablet for tablet 478e533f0fe049f7a110b79f5339f8c8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=134a4b9c14424ae188d91b77f058d927]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.462141 20296 tablet_service.cc:1467] Processing CreateTablet for tablet d77ddb854c034174babe1a753259cfc8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=134a4b9c14424ae188d91b77f058d927]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.462141 20564 tablet_service.cc:1467] Processing CreateTablet for tablet d77ddb854c034174babe1a753259cfc8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=134a4b9c14424ae188d91b77f058d927]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.462234 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77ddb854c034174babe1a753259cfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.462435 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77ddb854c034174babe1a753259cfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.462436 20564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77ddb854c034174babe1a753259cfc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.462540 20025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478e533f0fe049f7a110b79f5339f8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.463836 20295 tablet_service.cc:1467] Processing CreateTablet for tablet 478e533f0fe049f7a110b79f5339f8c8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=134a4b9c14424ae188d91b77f058d927]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.464578 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478e533f0fe049f7a110b79f5339f8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.464815 21006 tablet_bootstrap.cc:492] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:35.465950 21006 tablet_bootstrap.cc:654] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.466212 21010 tablet_bootstrap.cc:492] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:35.466899 20155 tablet_service.cc:1467] Processing CreateTablet for tablet 2b69fd1c5a404d2fbe5d3b2f56f09556 (DEFAULT_TABLE table=rebalance_test_table_5 [id=134a4b9c14424ae188d91b77f058d927]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.467214 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b69fd1c5a404d2fbe5d3b2f56f09556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.467212 21010 tablet_bootstrap.cc:654] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.468102 20432 tablet_service.cc:1467] Processing CreateTablet for tablet 2b69fd1c5a404d2fbe5d3b2f56f09556 (DEFAULT_TABLE table=rebalance_test_table_5 [id=134a4b9c14424ae188d91b77f058d927]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.468436 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b69fd1c5a404d2fbe5d3b2f56f09556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.468541 21010 tablet_bootstrap.cc:492] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:35.468644 21010 ts_tablet_manager.cc:1367] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:35.469225 21010 raft_consensus.cc:348] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:35.469341 21010 raft_consensus.cc:374] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.469372 21010 raft_consensus.cc:729] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.469511 21010 consensus_queue.cc:260] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:35.469889 21010 ts_tablet_manager.cc:1398] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:35.470062 21010 tablet_bootstrap.cc:492] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:35.470299 21006 tablet_bootstrap.cc:492] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:35.470414 21006 ts_tablet_manager.cc:1367] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:45:35.470834 21007 tablet_bootstrap.cc:492] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:35.470932 21006 raft_consensus.cc:348] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:35.471038 21006 raft_consensus.cc:374] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.471079 21006 raft_consensus.cc:729] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.471441 21011 tablet_bootstrap.cc:492] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:35.471563 21010 tablet_bootstrap.cc:654] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.471855 21007 tablet_bootstrap.cc:654] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.472317 21011 tablet_bootstrap.cc:654] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.473049 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 2b69fd1c5a404d2fbe5d3b2f56f09556 (DEFAULT_TABLE table=rebalance_test_table_5 [id=134a4b9c14424ae188d91b77f058d927]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.473419 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b69fd1c5a404d2fbe5d3b2f56f09556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.473875 21011 tablet_bootstrap.cc:492] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:35.473968 21011 ts_tablet_manager.cc:1367] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:45:35.474349 21007 tablet_bootstrap.cc:492] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:35.474435 21007 ts_tablet_manager.cc:1367] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:35.474491 21011 raft_consensus.cc:348] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.474617 21011 raft_consensus.cc:374] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.474660 21011 raft_consensus.cc:729] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.474771 21004 tablet_bootstrap.cc:492] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:35.474799 21011 consensus_queue.cc:260] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.474993 21007 raft_consensus.cc:348] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:35.475100 21007 raft_consensus.cc:374] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.475133 21007 raft_consensus.cc:729] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.475196 21011 ts_tablet_manager.cc:1398] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:45:35.475245 21007 consensus_queue.cc:260] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:35.475523 21007 ts_tablet_manager.cc:1398] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:35.475735 21007 tablet_bootstrap.cc:492] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:35.475756 21004 tablet_bootstrap.cc:654] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.476018 21003 tablet_bootstrap.cc:492] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:35.476966 21003 tablet_bootstrap.cc:654] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.476981 21004 tablet_bootstrap.cc:492] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:35.477149 21004 ts_tablet_manager.cc:1367] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:35.477699 21004 raft_consensus.cc:348] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.477813 21004 raft_consensus.cc:374] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.477844 21004 raft_consensus.cc:729] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.477984 21004 consensus_queue.cc:260] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.478041 20564 tablet_service.cc:1467] Processing CreateTablet for tablet 478e533f0fe049f7a110b79f5339f8c8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=134a4b9c14424ae188d91b77f058d927]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:35.478282 21004 ts_tablet_manager.cc:1398] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:35.478324 20564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478e533f0fe049f7a110b79f5339f8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:35.480984 21007 tablet_bootstrap.cc:654] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.481357 21006 consensus_queue.cc:260] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:35.481547 21003 tablet_bootstrap.cc:492] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:35.481648 21003 ts_tablet_manager.cc:1367] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:45:35.481756 21006 ts_tablet_manager.cc:1398] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.011s	user 0.001s	sys 0.000s
I20240502 20:45:35.482131 21003 raft_consensus.cc:348] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.482244 21003 raft_consensus.cc:374] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.482276 21003 raft_consensus.cc:729] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.482596 21006 tablet_bootstrap.cc:492] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:35.483004 21003 consensus_queue.cc:260] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.483130 21010 tablet_bootstrap.cc:492] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:35.483244 21010 ts_tablet_manager.cc:1367] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.000s
I20240502 20:45:35.483772 21010 raft_consensus.cc:348] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:35.483896 21010 raft_consensus.cc:374] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.483898 21003 ts_tablet_manager.cc:1398] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:35.483932 21010 raft_consensus.cc:729] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.484069 21010 consensus_queue.cc:260] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:35.484316 21010 ts_tablet_manager.cc:1398] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:35.484395 21006 tablet_bootstrap.cc:654] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:35.485970 21007 tablet_bootstrap.cc:492] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:35.486085 21007 ts_tablet_manager.cc:1367] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 20:45:35.486564 21007 raft_consensus.cc:348] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:35.486719 21007 raft_consensus.cc:374] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.486764 21007 raft_consensus.cc:729] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.486896 21007 consensus_queue.cc:260] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:35.487581 21006 tablet_bootstrap.cc:492] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:35.487731 21006 ts_tablet_manager.cc:1367] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:35.487818 21007 ts_tablet_manager.cc:1398] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:35.488301 21006 raft_consensus.cc:348] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:35.488451 21006 raft_consensus.cc:374] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:35.488493 21006 raft_consensus.cc:729] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:35.488658 21006 consensus_queue.cc:260] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:35.488976 21006 ts_tablet_manager.cc:1398] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:35.493945 21024 raft_consensus.cc:482] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:35.494071 21024 raft_consensus.cc:504] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:35.494501 21024 leader_election.cc:288] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:35.494863 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478e533f0fe049f7a110b79f5339f8c8" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:35.495021 20316 raft_consensus.cc:2454] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 0.
I20240502 20:45:35.495061 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478e533f0fe049f7a110b79f5339f8c8" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8" is_pre_election: true
I20240502 20:45:35.495227 20045 raft_consensus.cc:2454] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 0.
I20240502 20:45:35.495383 20518 leader_election.cc:302] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:35.495563 21024 raft_consensus.cc:2781] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:35.495640 21024 raft_consensus.cc:482] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:35.495684 21024 raft_consensus.cc:3037] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.496637 21024 raft_consensus.cc:504] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:35.497004 21024 leader_election.cc:288] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 election: Requested vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:35.497414 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478e533f0fe049f7a110b79f5339f8c8" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8"
I20240502 20:45:35.497395 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478e533f0fe049f7a110b79f5339f8c8" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:35.497543 20045 raft_consensus.cc:3037] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.497574 20316 raft_consensus.cc:3037] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.499231 20316 raft_consensus.cc:2454] T 478e533f0fe049f7a110b79f5339f8c8 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 1.
I20240502 20:45:35.499231 20045 raft_consensus.cc:2454] T 478e533f0fe049f7a110b79f5339f8c8 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 1.
I20240502 20:45:35.499593 20518 leader_election.cc:302] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:35.499838 21024 raft_consensus.cc:2781] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:35.499948 21024 raft_consensus.cc:686] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [term 1 LEADER]: Becoming Leader. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Running, Role: LEADER
I20240502 20:45:35.500092 21024 consensus_queue.cc:237] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:35.502081 19765 catalog_manager.cc:5484] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa7ea13bcb524353a03145dda1461c9e (127.19.66.134). New cstate: current_term: 1 leader_uuid: "fa7ea13bcb524353a03145dda1461c9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:35.600749 20766 raft_consensus.cc:482] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:35.601011 20766 raft_consensus.cc:504] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.601713 20766 leader_election.cc:288] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:35.602092 21066 raft_consensus.cc:482] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:35.602398 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69fd1c5a404d2fbe5d3b2f56f09556" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:35.602452 21066 raft_consensus.cc:504] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.602566 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69fd1c5a404d2fbe5d3b2f56f09556" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:35.602711 20450 raft_consensus.cc:2454] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:35.602841 19911 raft_consensus.cc:2454] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:35.603171 21066 leader_election.cc:288] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:35.603206 20112 leader_election.cc:302] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 
I20240502 20:45:35.603477 20766 raft_consensus.cc:2781] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:35.603549 20766 raft_consensus.cc:482] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:35.603539 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69fd1c5a404d2fbe5d3b2f56f09556" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:35.603603 20766 raft_consensus.cc:3037] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.603667 20450 raft_consensus.cc:2454] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:35.603713 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69fd1c5a404d2fbe5d3b2f56f09556" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" is_pre_election: true
I20240502 20:45:35.604180 19843 leader_election.cc:302] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:35.604475 21066 raft_consensus.cc:2781] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:35.604543 21066 raft_consensus.cc:482] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:35.604573 21066 raft_consensus.cc:3037] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.605620 21066 raft_consensus.cc:504] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.605803 20766 raft_consensus.cc:504] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.606029 21066 leader_election.cc:288] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Requested vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:35.606101 20179 raft_consensus.cc:2379] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91942b13253244aab62c2a52e1fdd827 in current term 1: Already voted for candidate 900c16672cb34468bbd3e8d3e77ddc1d in this term.
I20240502 20:45:35.606290 20766 leader_election.cc:288] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Requested vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:35.606554 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69fd1c5a404d2fbe5d3b2f56f09556" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:35.606670 20450 raft_consensus.cc:3037] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.606706 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69fd1c5a404d2fbe5d3b2f56f09556" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:35.606683 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69fd1c5a404d2fbe5d3b2f56f09556" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:35.606859 19911 raft_consensus.cc:2379] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 900c16672cb34468bbd3e8d3e77ddc1d in current term 1: Already voted for candidate 91942b13253244aab62c2a52e1fdd827 in this term.
I20240502 20:45:35.607092 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69fd1c5a404d2fbe5d3b2f56f09556" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
I20240502 20:45:35.607223 20179 raft_consensus.cc:2379] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 91942b13253244aab62c2a52e1fdd827 in current term 1: Already voted for candidate 900c16672cb34468bbd3e8d3e77ddc1d in this term.
I20240502 20:45:35.607199 20112 leader_election.cc:302] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827
I20240502 20:45:35.607759 20766 raft_consensus.cc:2726] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 20:45:35.607868 20450 raft_consensus.cc:2454] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:35.608605 19843 leader_election.cc:302] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 900c16672cb34468bbd3e8d3e77ddc1d
I20240502 20:45:35.608806 21066 raft_consensus.cc:2781] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:35.608897 21066 raft_consensus.cc:686] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [term 1 LEADER]: Becoming Leader. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Running, Role: LEADER
I20240502 20:45:35.609050 21066 consensus_queue.cc:237] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:35.610994 19765 catalog_manager.cc:5484] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91942b13253244aab62c2a52e1fdd827 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "91942b13253244aab62c2a52e1fdd827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:35.708122 20900 raft_consensus.cc:482] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:35.708349 20900 raft_consensus.cc:504] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:35.708925 20900 leader_election.cc:288] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393), c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391)
I20240502 20:45:35.709789 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77ddb854c034174babe1a753259cfc8" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:35.709782 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77ddb854c034174babe1a753259cfc8" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:35.709977 20316 raft_consensus.cc:2454] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:35.709983 20585 raft_consensus.cc:2454] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:35.710351 19979 leader_election.cc:302] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:35.710558 20900 raft_consensus.cc:2781] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:35.710633 20900 raft_consensus.cc:482] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:35.710676 20900 raft_consensus.cc:3037] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.711824 20900 raft_consensus.cc:504] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:35.712221 20900 leader_election.cc:288] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Requested vote from peers fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393), c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391)
I20240502 20:45:35.712885 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77ddb854c034174babe1a753259cfc8" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:35.712885 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77ddb854c034174babe1a753259cfc8" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:35.713120 20316 raft_consensus.cc:3037] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.713120 20585 raft_consensus.cc:3037] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:35.714262 20316 raft_consensus.cc:2454] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:35.714262 20585 raft_consensus.cc:2454] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:35.714684 19979 leader_election.cc:302] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:35.714927 20900 raft_consensus.cc:2781] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:35.715348 20900 raft_consensus.cc:686] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [term 1 LEADER]: Becoming Leader. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Running, Role: LEADER
I20240502 20:45:35.715559 20900 consensus_queue.cc:237] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:35.717640 19765 catalog_manager.cc:5484] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951291b2e23f46a79812e404a06690d8 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "951291b2e23f46a79812e404a06690d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:35.898767 21024 consensus_queue.cc:1035] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:35.902396 21024 consensus_queue.cc:1035] T 478e533f0fe049f7a110b79f5339f8c8 P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:45:35.910748 21087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:35.911036 21087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:35.914520 21087 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 20:45:36.010035 20316 raft_consensus.cc:1261] T d77ddb854c034174babe1a753259cfc8 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.010510 20585 raft_consensus.cc:1261] T d77ddb854c034174babe1a753259cfc8 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.011313 20900 consensus_queue.cc:1035] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:36.011489 21085 consensus_queue.cc:1035] T d77ddb854c034174babe1a753259cfc8 P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:36.051744 20179 raft_consensus.cc:1261] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.052386 21066 consensus_queue.cc:1035] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:36.053639 20450 raft_consensus.cc:1261] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.054456 21066 consensus_queue.cc:1035] T 2b69fd1c5a404d2fbe5d3b2f56f09556 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 120.108 ms
  time per row: 0.0600542 ms
I20240502 20:45:36.088253 19763 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:36.096228 20432 tablet_service.cc:1467] Processing CreateTablet for tablet 3b08134c7baf48739d2464fc1de7d943 (DEFAULT_TABLE table=rebalance_test_table_6 [id=10d3e338d4be4943826b58256fdbf134]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.096309 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 3b08134c7baf48739d2464fc1de7d943 (DEFAULT_TABLE table=rebalance_test_table_6 [id=10d3e338d4be4943826b58256fdbf134]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.096662 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b08134c7baf48739d2464fc1de7d943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.096671 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b08134c7baf48739d2464fc1de7d943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.097131 20155 tablet_service.cc:1467] Processing CreateTablet for tablet 3b08134c7baf48739d2464fc1de7d943 (DEFAULT_TABLE table=rebalance_test_table_6 [id=10d3e338d4be4943826b58256fdbf134]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.097288 20295 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7fda326b754878956cddd3bd5d6037 (DEFAULT_TABLE table=rebalance_test_table_6 [id=10d3e338d4be4943826b58256fdbf134]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.097471 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b08134c7baf48739d2464fc1de7d943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.097568 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7fda326b754878956cddd3bd5d6037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.099614 21162 tablet_bootstrap.cc:492] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:36.100549 21163 tablet_bootstrap.cc:492] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:36.101541 19891 tablet_service.cc:1467] Processing CreateTablet for tablet e4cb9bf04d954ac4a16e71128cbf991a (DEFAULT_TABLE table=rebalance_test_table_6 [id=10d3e338d4be4943826b58256fdbf134]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.101603 20564 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7fda326b754878956cddd3bd5d6037 (DEFAULT_TABLE table=rebalance_test_table_6 [id=10d3e338d4be4943826b58256fdbf134]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.101853 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cb9bf04d954ac4a16e71128cbf991a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.101933 20564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7fda326b754878956cddd3bd5d6037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.101954 21164 tablet_bootstrap.cc:492] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:36.102329 21163 tablet_bootstrap.cc:654] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.103358 21164 tablet_bootstrap.cc:654] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.101240 21162 tablet_bootstrap.cc:654] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.104688 20026 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7fda326b754878956cddd3bd5d6037 (DEFAULT_TABLE table=rebalance_test_table_6 [id=10d3e338d4be4943826b58256fdbf134]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.105031 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7fda326b754878956cddd3bd5d6037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.105612 21164 tablet_bootstrap.cc:492] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:36.105692 21164 ts_tablet_manager.cc:1367] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:36.105870 21162 tablet_bootstrap.cc:492] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:36.105959 21162 ts_tablet_manager.cc:1367] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.002s
I20240502 20:45:36.106207 21164 raft_consensus.cc:348] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:36.106350 21164 raft_consensus.cc:374] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.106385 21164 raft_consensus.cc:729] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.106520 21164 consensus_queue.cc:260] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:36.106806 21164 ts_tablet_manager.cc:1398] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.106541 21162 raft_consensus.cc:348] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.107506 21162 raft_consensus.cc:374] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.107544 21162 raft_consensus.cc:729] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.107695 21162 consensus_queue.cc:260] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.108000 21162 ts_tablet_manager.cc:1398] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 20:45:36.108215 21162 tablet_bootstrap.cc:492] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:36.108389 20155 tablet_service.cc:1467] Processing CreateTablet for tablet e4cb9bf04d954ac4a16e71128cbf991a (DEFAULT_TABLE table=rebalance_test_table_6 [id=10d3e338d4be4943826b58256fdbf134]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.108687 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cb9bf04d954ac4a16e71128cbf991a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.109186 20432 tablet_service.cc:1467] Processing CreateTablet for tablet e4cb9bf04d954ac4a16e71128cbf991a (DEFAULT_TABLE table=rebalance_test_table_6 [id=10d3e338d4be4943826b58256fdbf134]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.109387 21162 tablet_bootstrap.cc:654] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.109421 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cb9bf04d954ac4a16e71128cbf991a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.110774 21162 tablet_bootstrap.cc:492] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:36.110852 21162 ts_tablet_manager.cc:1367] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:45:36.111450 21162 raft_consensus.cc:348] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.111599 21162 raft_consensus.cc:374] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.111673 21162 raft_consensus.cc:729] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.111894 21162 consensus_queue.cc:260] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.112864 21169 tablet_bootstrap.cc:492] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:36.112911 21162 ts_tablet_manager.cc:1398] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 20:45:36.112973 21166 tablet_bootstrap.cc:492] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:36.115309 21166 tablet_bootstrap.cc:654] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.115896 21169 tablet_bootstrap.cc:654] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.116288 21161 tablet_bootstrap.cc:492] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:36.117411 21161 tablet_bootstrap.cc:654] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.117971 21163 tablet_bootstrap.cc:492] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:36.118098 21163 ts_tablet_manager.cc:1367] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.018s	user 0.002s	sys 0.000s
I20240502 20:45:36.118347 21166 tablet_bootstrap.cc:492] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:36.118443 21166 ts_tablet_manager.cc:1367] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.000s
I20240502 20:45:36.118499 21161 tablet_bootstrap.cc:492] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:36.118499 21169 tablet_bootstrap.cc:492] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:36.118583 21161 ts_tablet_manager.cc:1367] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:36.118584 21169 ts_tablet_manager.cc:1367] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:45:36.118636 21163 raft_consensus.cc:348] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.118820 21163 raft_consensus.cc:374] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.118858 21163 raft_consensus.cc:729] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.118963 21166 raft_consensus.cc:348] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:36.119016 21163 consensus_queue.cc:260] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.119071 21166 raft_consensus.cc:374] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.119112 21166 raft_consensus.cc:729] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.119145 21169 raft_consensus.cc:348] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:36.119179 21161 raft_consensus.cc:348] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.119262 21169 raft_consensus.cc:374] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.119285 21161 raft_consensus.cc:374] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.119295 21169 raft_consensus.cc:729] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.119323 21161 raft_consensus.cc:729] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.119282 21166 consensus_queue.cc:260] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:36.119366 21163 ts_tablet_manager.cc:1398] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.119460 21161 consensus_queue.cc:260] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.119460 21169 consensus_queue.cc:260] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:36.119575 21163 tablet_bootstrap.cc:492] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:36.119688 21166 ts_tablet_manager.cc:1398] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.119736 21161 ts_tablet_manager.cc:1398] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.119808 21169 ts_tablet_manager.cc:1398] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.119927 21161 tablet_bootstrap.cc:492] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:36.120609 21163 tablet_bootstrap.cc:654] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.120651 21161 tablet_bootstrap.cc:654] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.122288 21161 tablet_bootstrap.cc:492] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:36.122323 21163 tablet_bootstrap.cc:492] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:36.122387 21161 ts_tablet_manager.cc:1367] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 20:45:36.122416 21163 ts_tablet_manager.cc:1367] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:36.122831 21161 raft_consensus.cc:348] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.122963 21161 raft_consensus.cc:374] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.122926 21163 raft_consensus.cc:348] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.123006 21161 raft_consensus.cc:729] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.123045 21163 raft_consensus.cc:374] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.123083 21163 raft_consensus.cc:729] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.123144 21161 consensus_queue.cc:260] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.123215 21163 consensus_queue.cc:260] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.123445 21161 ts_tablet_manager.cc:1398] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:45:36.123598 21163 ts_tablet_manager.cc:1398] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.141587 21089 raft_consensus.cc:482] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:36.141731 21089 raft_consensus.cc:504] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:36.142189 21089 leader_election.cc:288] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:36.142652 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7fda326b754878956cddd3bd5d6037" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:36.142707 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7fda326b754878956cddd3bd5d6037" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8" is_pre_election: true
I20240502 20:45:36.142805 20316 raft_consensus.cc:2454] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 0.
I20240502 20:45:36.142844 20045 raft_consensus.cc:2454] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 0.
I20240502 20:45:36.143172 20518 leader_election.cc:302] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:36.143395 21089 raft_consensus.cc:2781] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:36.143458 21089 raft_consensus.cc:482] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:36.143487 21089 raft_consensus.cc:3037] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.144424 21089 raft_consensus.cc:504] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:36.144821 21089 leader_election.cc:288] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 election: Requested vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:36.145324 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7fda326b754878956cddd3bd5d6037" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8"
I20240502 20:45:36.145473 20045 raft_consensus.cc:3037] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.145473 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7fda326b754878956cddd3bd5d6037" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:36.145706 20316 raft_consensus.cc:3037] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.146405 20045 raft_consensus.cc:2454] T 9f7fda326b754878956cddd3bd5d6037 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 1.
I20240502 20:45:36.146857 20316 raft_consensus.cc:2454] T 9f7fda326b754878956cddd3bd5d6037 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 1.
I20240502 20:45:36.146831 20520 leader_election.cc:302] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:36.147281 21089 raft_consensus.cc:2781] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:36.147420 21089 raft_consensus.cc:686] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [term 1 LEADER]: Becoming Leader. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Running, Role: LEADER
I20240502 20:45:36.147612 21089 consensus_queue.cc:237] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:36.150322 19767 catalog_manager.cc:5484] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa7ea13bcb524353a03145dda1461c9e (127.19.66.134). New cstate: current_term: 1 leader_uuid: "fa7ea13bcb524353a03145dda1461c9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:36.203161 20766 raft_consensus.cc:482] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:36.203366 20766 raft_consensus.cc:504] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.203882 20766 leader_election.cc:288] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:36.204465 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b08134c7baf48739d2464fc1de7d943" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:36.204573 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b08134c7baf48739d2464fc1de7d943" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:36.204756 19911 raft_consensus.cc:2454] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:36.204831 20450 raft_consensus.cc:2454] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:36.205261 20112 leader_election.cc:302] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:36.205612 20766 raft_consensus.cc:2781] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:36.205698 20766 raft_consensus.cc:482] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:36.205737 20766 raft_consensus.cc:3037] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.206633 20766 raft_consensus.cc:504] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.206975 20766 leader_election.cc:288] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Requested vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:36.207473 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b08134c7baf48739d2464fc1de7d943" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:36.207473 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b08134c7baf48739d2464fc1de7d943" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:36.207609 19911 raft_consensus.cc:3037] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.207616 20450 raft_consensus.cc:3037] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.208532 20450 raft_consensus.cc:2454] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:36.208573 19911 raft_consensus.cc:2454] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:36.208962 20112 leader_election.cc:302] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:36.209251 20766 raft_consensus.cc:2781] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:36.209473 20766 raft_consensus.cc:686] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 LEADER]: Becoming Leader. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Running, Role: LEADER
I20240502 20:45:36.209635 20766 consensus_queue.cc:237] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.211504 19767 catalog_manager.cc:5484] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131). New cstate: current_term: 1 leader_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:36.343565 21068 raft_consensus.cc:482] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:36.343751 21068 raft_consensus.cc:504] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.344233 21068 leader_election.cc:288] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:36.344769 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cb9bf04d954ac4a16e71128cbf991a" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:36.344769 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cb9bf04d954ac4a16e71128cbf991a" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" is_pre_election: true
I20240502 20:45:36.344905 19911 raft_consensus.cc:2454] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 0.
I20240502 20:45:36.344933 20179 raft_consensus.cc:2454] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 0.
I20240502 20:45:36.345266 20385 leader_election.cc:302] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:36.345464 21068 raft_consensus.cc:2781] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:36.345525 21068 raft_consensus.cc:482] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:36.345564 21068 raft_consensus.cc:3037] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.346833 21068 raft_consensus.cc:504] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.347255 21068 leader_election.cc:288] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 election: Requested vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:36.347663 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cb9bf04d954ac4a16e71128cbf991a" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
I20240502 20:45:36.347795 20179 raft_consensus.cc:3037] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.347743 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cb9bf04d954ac4a16e71128cbf991a" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:36.347900 19911 raft_consensus.cc:3037] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.348656 20179 raft_consensus.cc:2454] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 1.
I20240502 20:45:36.348716 19911 raft_consensus.cc:2454] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 1.
I20240502 20:45:36.348980 20386 leader_election.cc:302] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 
I20240502 20:45:36.349231 21068 raft_consensus.cc:2781] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:36.349462 21068 raft_consensus.cc:686] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [term 1 LEADER]: Becoming Leader. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Running, Role: LEADER
I20240502 20:45:36.349624 21068 consensus_queue.cc:237] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:36.351545 19767 catalog_manager.cc:5484] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5742a696d4c3406086b710e571e2cf0e (127.19.66.133). New cstate: current_term: 1 leader_uuid: "5742a696d4c3406086b710e571e2cf0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:36.536881 21182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:36.537209 21182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:36.540850 21182 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 20:45:36.579948 21089 consensus_queue.cc:1035] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:36.584654 21089 consensus_queue.cc:1035] T 9f7fda326b754878956cddd3bd5d6037 P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:36.653807 20179 raft_consensus.cc:1261] T e4cb9bf04d954ac4a16e71128cbf991a P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Refusing update from remote peer 5742a696d4c3406086b710e571e2cf0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.654302 21068 consensus_queue.cc:1035] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:36.657027 19911 raft_consensus.cc:1261] T e4cb9bf04d954ac4a16e71128cbf991a P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Refusing update from remote peer 5742a696d4c3406086b710e571e2cf0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.658360 21180 consensus_queue.cc:1035] T e4cb9bf04d954ac4a16e71128cbf991a P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:36.665714 19910 raft_consensus.cc:1261] T 3b08134c7baf48739d2464fc1de7d943 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.666163 20450 raft_consensus.cc:1261] T 3b08134c7baf48739d2464fc1de7d943 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:36.666813 20766 consensus_queue.cc:1035] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:36.667138 20766 consensus_queue.cc:1035] T 3b08134c7baf48739d2464fc1de7d943 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 126.294 ms
  time per row: 0.0631468 ms
I20240502 20:45:36.727291 19763 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:36.734457 20295 tablet_service.cc:1467] Processing CreateTablet for tablet 768868443e13485398a1c3e23a005083 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7e9686eda662416a84c55afd906ded09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.734699 19891 tablet_service.cc:1467] Processing CreateTablet for tablet b56e5eee0c344c03a74f3bb0e62bb8d1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7e9686eda662416a84c55afd906ded09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.735000 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56e5eee0c344c03a74f3bb0e62bb8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.734946 20432 tablet_service.cc:1467] Processing CreateTablet for tablet b56e5eee0c344c03a74f3bb0e62bb8d1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7e9686eda662416a84c55afd906ded09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.735244 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56e5eee0c344c03a74f3bb0e62bb8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.737337 21254 tablet_bootstrap.cc:492] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:36.738116 20155 tablet_service.cc:1467] Processing CreateTablet for tablet b56e5eee0c344c03a74f3bb0e62bb8d1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7e9686eda662416a84c55afd906ded09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.738381 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56e5eee0c344c03a74f3bb0e62bb8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.738565 21254 tablet_bootstrap.cc:654] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.738880 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 85eb80e5bd9b49109210d7fb915d3e49 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7e9686eda662416a84c55afd906ded09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.739509 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768868443e13485398a1c3e23a005083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.740526 21256 tablet_bootstrap.cc:492] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:36.741443 21256 tablet_bootstrap.cc:654] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.741703 20564 tablet_service.cc:1467] Processing CreateTablet for tablet 768868443e13485398a1c3e23a005083 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7e9686eda662416a84c55afd906ded09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.741779 20026 tablet_service.cc:1467] Processing CreateTablet for tablet 768868443e13485398a1c3e23a005083 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7e9686eda662416a84c55afd906ded09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.741956 20564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768868443e13485398a1c3e23a005083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.742049 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768868443e13485398a1c3e23a005083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.743016 20565 tablet_service.cc:1467] Processing CreateTablet for tablet 85eb80e5bd9b49109210d7fb915d3e49 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7e9686eda662416a84c55afd906ded09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.743235 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85eb80e5bd9b49109210d7fb915d3e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.744155 21258 tablet_bootstrap.cc:492] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:36.744421 20025 tablet_service.cc:1467] Processing CreateTablet for tablet 85eb80e5bd9b49109210d7fb915d3e49 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7e9686eda662416a84c55afd906ded09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:36.744648 20025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85eb80e5bd9b49109210d7fb915d3e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.745059 21258 tablet_bootstrap.cc:654] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.745782 21256 tablet_bootstrap.cc:492] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:36.745877 21256 ts_tablet_manager.cc:1367] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:36.746331 21256 raft_consensus.cc:348] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:36.746441 21256 raft_consensus.cc:374] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.746472 21256 raft_consensus.cc:729] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.746636 21256 consensus_queue.cc:260] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:36.747287 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85eb80e5bd9b49109210d7fb915d3e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:36.747467 21256 ts_tablet_manager.cc:1398] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:36.747728 21258 tablet_bootstrap.cc:492] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:36.747804 21258 ts_tablet_manager.cc:1367] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.000s
I20240502 20:45:36.748239 21258 raft_consensus.cc:348] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.748347 21258 raft_consensus.cc:374] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.748389 21258 raft_consensus.cc:729] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.748515 21258 consensus_queue.cc:260] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.748808 21258 ts_tablet_manager.cc:1398] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.749599 21258 tablet_bootstrap.cc:492] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:36.750523 21258 tablet_bootstrap.cc:654] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.751752 21258 tablet_bootstrap.cc:492] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:36.752074 21263 tablet_bootstrap.cc:492] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:36.752344 21259 tablet_bootstrap.cc:492] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:36.752379 21258 ts_tablet_manager.cc:1367] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:36.752961 21258 raft_consensus.cc:348] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.753185 21258 raft_consensus.cc:374] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.753271 21258 raft_consensus.cc:729] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.753600 21258 consensus_queue.cc:260] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.753743 21259 tablet_bootstrap.cc:654] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.754455 21258 ts_tablet_manager.cc:1398] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:36.754782 21261 tablet_bootstrap.cc:492] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:36.756315 21263 tablet_bootstrap.cc:654] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.756592 21254 tablet_bootstrap.cc:492] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:36.756704 21254 ts_tablet_manager.cc:1367] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.019s	user 0.003s	sys 0.000s
I20240502 20:45:36.757032 21259 tablet_bootstrap.cc:492] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:36.757167 21259 ts_tablet_manager.cc:1367] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:36.757331 21261 tablet_bootstrap.cc:654] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.757258 21254 raft_consensus.cc:348] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:36.757386 21254 raft_consensus.cc:374] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.757431 21254 raft_consensus.cc:729] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.757586 21254 consensus_queue.cc:260] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:36.757761 21259 raft_consensus.cc:348] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.757874 21259 raft_consensus.cc:374] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.757898 21254 ts_tablet_manager.cc:1398] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.757906 21259 raft_consensus.cc:729] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.758070 21259 consensus_queue.cc:260] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.758525 21259 ts_tablet_manager.cc:1398] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.759023 21259 tablet_bootstrap.cc:492] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:36.759203 21261 tablet_bootstrap.cc:492] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:36.759317 21261 ts_tablet_manager.cc:1367] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:36.759394 21263 tablet_bootstrap.cc:492] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:36.759508 21263 ts_tablet_manager.cc:1367] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 20:45:36.759825 21261 raft_consensus.cc:348] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.759951 21261 raft_consensus.cc:374] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.759994 21261 raft_consensus.cc:729] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.759964 21263 raft_consensus.cc:348] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:36.760087 21263 raft_consensus.cc:374] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.760130 21263 raft_consensus.cc:729] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.760124 21261 consensus_queue.cc:260] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.760277 21263 consensus_queue.cc:260] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:36.760496 21261 ts_tablet_manager.cc:1398] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.760556 21263 ts_tablet_manager.cc:1398] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:36.761265 21261 tablet_bootstrap.cc:492] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:36.761966 21259 tablet_bootstrap.cc:654] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.763011 21261 tablet_bootstrap.cc:654] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:36.763407 21259 tablet_bootstrap.cc:492] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:36.763516 21259 ts_tablet_manager.cc:1367] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:36.763990 21259 raft_consensus.cc:348] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.764122 21259 raft_consensus.cc:374] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.764169 21259 raft_consensus.cc:729] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.764302 21259 consensus_queue.cc:260] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.764408 21261 tablet_bootstrap.cc:492] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:36.764525 21261 ts_tablet_manager.cc:1367] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:36.764621 21259 ts_tablet_manager.cc:1398] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:36.765040 21261 raft_consensus.cc:348] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.765208 21261 raft_consensus.cc:374] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:36.765290 21261 raft_consensus.cc:729] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:36.765455 21261 consensus_queue.cc:260] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.765746 21261 ts_tablet_manager.cc:1398] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:45:36.785622 21068 raft_consensus.cc:482] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:36.785765 21068 raft_consensus.cc:504] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:36.786196 21068 leader_election.cc:288] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:36.786732 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56e5eee0c344c03a74f3bb0e62bb8d1" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" is_pre_election: true
I20240502 20:45:36.786734 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56e5eee0c344c03a74f3bb0e62bb8d1" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:36.786876 20179 raft_consensus.cc:2454] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 0.
I20240502 20:45:36.786900 19911 raft_consensus.cc:2454] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 0.
I20240502 20:45:36.787231 20385 leader_election.cc:302] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:36.787467 21068 raft_consensus.cc:2781] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:36.787525 21068 raft_consensus.cc:482] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:36.787580 21068 raft_consensus.cc:3037] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.788508 21068 raft_consensus.cc:504] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:36.788852 21068 leader_election.cc:288] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 election: Requested vote from peers 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:36.789268 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56e5eee0c344c03a74f3bb0e62bb8d1" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:36.789270 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56e5eee0c344c03a74f3bb0e62bb8d1" candidate_uuid: "5742a696d4c3406086b710e571e2cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
I20240502 20:45:36.789395 19911 raft_consensus.cc:3037] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.789424 20179 raft_consensus.cc:3037] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.790252 20179 raft_consensus.cc:2454] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 1.
I20240502 20:45:36.790557 20386 leader_election.cc:302] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 
I20240502 20:45:36.790707 19911 raft_consensus.cc:2454] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5742a696d4c3406086b710e571e2cf0e in term 1.
I20240502 20:45:36.790731 21068 raft_consensus.cc:2781] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:36.790829 21068 raft_consensus.cc:686] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [term 1 LEADER]: Becoming Leader. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Running, Role: LEADER
I20240502 20:45:36.790979 21068 consensus_queue.cc:237] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:36.792846 19763 catalog_manager.cc:5484] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5742a696d4c3406086b710e571e2cf0e (127.19.66.133). New cstate: current_term: 1 leader_uuid: "5742a696d4c3406086b710e571e2cf0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:36.864622 21212 raft_consensus.cc:482] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:36.864795 21212 raft_consensus.cc:504] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.865300 21212 leader_election.cc:288] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:36.865850 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768868443e13485398a1c3e23a005083" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8" is_pre_election: true
I20240502 20:45:36.865895 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768868443e13485398a1c3e23a005083" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:36.866012 20045 raft_consensus.cc:2454] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:36.866021 20585 raft_consensus.cc:2454] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:36.866328 20249 leader_election.cc:302] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:36.866490 21212 raft_consensus.cc:2781] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:36.866554 21212 raft_consensus.cc:482] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:36.866621 21212 raft_consensus.cc:3037] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.867532 21212 raft_consensus.cc:504] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.867877 21212 leader_election.cc:288] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Requested vote from peers 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:36.868276 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768868443e13485398a1c3e23a005083" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8"
I20240502 20:45:36.868325 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768868443e13485398a1c3e23a005083" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:36.868398 20045 raft_consensus.cc:3037] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.868444 20585 raft_consensus.cc:3037] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:36.869257 20045 raft_consensus.cc:2454] T 768868443e13485398a1c3e23a005083 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:36.869402 20585 raft_consensus.cc:2454] T 768868443e13485398a1c3e23a005083 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:36.869585 20249 leader_election.cc:302] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:36.869751 21212 raft_consensus.cc:2781] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:36.869845 21212 raft_consensus.cc:686] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [term 1 LEADER]: Becoming Leader. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Running, Role: LEADER
I20240502 20:45:36.869997 21212 consensus_queue.cc:237] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:36.871867 19763 catalog_manager.cc:5484] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af reported cstate change: term changed from 0 to 1, leader changed from <none> to c302712f8dbd4fa7bed797758fd772af (127.19.66.132). New cstate: current_term: 1 leader_uuid: "c302712f8dbd4fa7bed797758fd772af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:37.145622 21085 raft_consensus.cc:482] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:37.145758 21085 raft_consensus.cc:504] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:37.146171 21085 leader_election.cc:288] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:37.146617 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85eb80e5bd9b49109210d7fb915d3e49" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:37.146617 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85eb80e5bd9b49109210d7fb915d3e49" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:37.146772 20585 raft_consensus.cc:2454] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:37.146772 20316 raft_consensus.cc:2454] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:37.147133 19979 leader_election.cc:302] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:37.147369 21085 raft_consensus.cc:2781] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:37.147430 21085 raft_consensus.cc:482] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:37.147457 21085 raft_consensus.cc:3037] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.151965 21085 raft_consensus.cc:504] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:37.152479 21085 leader_election.cc:288] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Requested vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:37.152982 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85eb80e5bd9b49109210d7fb915d3e49" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:37.152985 20585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85eb80e5bd9b49109210d7fb915d3e49" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:37.153151 20316 raft_consensus.cc:3037] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.153173 20585 raft_consensus.cc:3037] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.154395 20585 raft_consensus.cc:2454] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:37.154807 19981 leader_election.cc:302] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:37.155072 21085 raft_consensus.cc:2781] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:37.155182 20316 raft_consensus.cc:2454] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:37.155300 21085 raft_consensus.cc:686] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [term 1 LEADER]: Becoming Leader. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Running, Role: LEADER
I20240502 20:45:37.155687 21085 consensus_queue.cc:237] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:37.157987 19763 catalog_manager.cc:5484] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951291b2e23f46a79812e404a06690d8 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "951291b2e23f46a79812e404a06690d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:37.170656 21068 consensus_queue.cc:1035] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:37.175923 21274 consensus_queue.cc:1035] T b56e5eee0c344c03a74f3bb0e62bb8d1 P 5742a696d4c3406086b710e571e2cf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:45:37.354207 21276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:37.354560 21276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:37.358194 21276 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:45:37.365804 21212 consensus_queue.cc:1035] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:37.370779 21094 consensus_queue.cc:1035] T 768868443e13485398a1c3e23a005083 P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240502 20:45:37.487900 20584 raft_consensus.cc:1261] T 85eb80e5bd9b49109210d7fb915d3e49 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:37.489324 21271 consensus_queue.cc:1035] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:37.489499 20316 raft_consensus.cc:1261] T 85eb80e5bd9b49109210d7fb915d3e49 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:37.491161 21271 consensus_queue.cc:1035] T 85eb80e5bd9b49109210d7fb915d3e49 P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 133.605 ms
  time per row: 0.0668027 ms
I20240502 20:45:37.550442 19761 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:37.558559 19891 tablet_service.cc:1467] Processing CreateTablet for tablet b27c88db38324ae79f3bfe1a91117298 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7911100759bf416fa91b937d8d456c86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:37.558933 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27c88db38324ae79f3bfe1a91117298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:37.559284 20155 tablet_service.cc:1467] Processing CreateTablet for tablet b27c88db38324ae79f3bfe1a91117298 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7911100759bf416fa91b937d8d456c86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:37.559612 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27c88db38324ae79f3bfe1a91117298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:37.559746 20432 tablet_service.cc:1467] Processing CreateTablet for tablet b27c88db38324ae79f3bfe1a91117298 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7911100759bf416fa91b937d8d456c86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:37.560031 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27c88db38324ae79f3bfe1a91117298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:37.561373 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 2aabafdc9be0486291fbcac1b8e04609 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7911100759bf416fa91b937d8d456c86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:37.561662 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aabafdc9be0486291fbcac1b8e04609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:37.561661 20025 tablet_service.cc:1467] Processing CreateTablet for tablet ac8ed69f637141f7b279d8eaced8121f (DEFAULT_TABLE table=rebalance_test_table_8 [id=7911100759bf416fa91b937d8d456c86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:37.561926 20025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8ed69f637141f7b279d8eaced8121f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:37.564252 21339 tablet_bootstrap.cc:492] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:37.564875 20565 tablet_service.cc:1467] Processing CreateTablet for tablet ac8ed69f637141f7b279d8eaced8121f (DEFAULT_TABLE table=rebalance_test_table_8 [id=7911100759bf416fa91b937d8d456c86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:37.565253 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8ed69f637141f7b279d8eaced8121f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:37.565384 21339 tablet_bootstrap.cc:654] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:37.565788 20295 tablet_service.cc:1467] Processing CreateTablet for tablet ac8ed69f637141f7b279d8eaced8121f (DEFAULT_TABLE table=rebalance_test_table_8 [id=7911100759bf416fa91b937d8d456c86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:37.565863 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 2aabafdc9be0486291fbcac1b8e04609 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7911100759bf416fa91b937d8d456c86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:37.566071 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8ed69f637141f7b279d8eaced8121f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:37.566133 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aabafdc9be0486291fbcac1b8e04609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:37.568588 20155 tablet_service.cc:1467] Processing CreateTablet for tablet 2aabafdc9be0486291fbcac1b8e04609 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7911100759bf416fa91b937d8d456c86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:37.568953 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aabafdc9be0486291fbcac1b8e04609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:37.569166 21340 tablet_bootstrap.cc:492] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:37.569597 21337 tablet_bootstrap.cc:492] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:37.570184 21340 tablet_bootstrap.cc:654] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:37.570685 21337 tablet_bootstrap.cc:654] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:37.571117 21342 tablet_bootstrap.cc:492] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:37.572043 21342 tablet_bootstrap.cc:654] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:37.573386 21340 tablet_bootstrap.cc:492] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:37.573406 21342 tablet_bootstrap.cc:492] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:37.573521 21342 ts_tablet_manager.cc:1367] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 20:45:37.573607 21340 ts_tablet_manager.cc:1367] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.001s
I20240502 20:45:37.573619 21343 tablet_bootstrap.cc:492] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:37.574079 21342 raft_consensus.cc:348] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:37.574090 21340 raft_consensus.cc:348] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:37.574199 21342 raft_consensus.cc:374] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:37.574198 21340 raft_consensus.cc:374] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:37.574234 21342 raft_consensus.cc:729] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:37.574234 21340 raft_consensus.cc:729] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:37.574378 21342 consensus_queue.cc:260] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:37.574374 21340 consensus_queue.cc:260] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:37.574661 21342 ts_tablet_manager.cc:1398] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:45:37.574671 21340 ts_tablet_manager.cc:1398] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:37.575374 21340 tablet_bootstrap.cc:492] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:37.576236 21337 tablet_bootstrap.cc:492] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:37.576516 21337 ts_tablet_manager.cc:1367] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.002s
I20240502 20:45:37.576908 21340 tablet_bootstrap.cc:654] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:37.577224 21337 raft_consensus.cc:348] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:37.577553 21337 raft_consensus.cc:374] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:37.577656 21337 raft_consensus.cc:729] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:37.577888 21337 consensus_queue.cc:260] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:37.578496 21337 ts_tablet_manager.cc:1398] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:37.578505 21343 tablet_bootstrap.cc:654] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:37.578769 21337 tablet_bootstrap.cc:492] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:37.579638 21338 tablet_bootstrap.cc:492] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:37.579943 21337 tablet_bootstrap.cc:654] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:37.580507 21343 tablet_bootstrap.cc:492] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:37.580637 21343 ts_tablet_manager.cc:1367] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.002s
I20240502 20:45:37.581208 21343 raft_consensus.cc:348] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:37.581341 21343 raft_consensus.cc:374] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:37.581375 21343 raft_consensus.cc:729] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:37.581552 21343 consensus_queue.cc:260] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:37.582043 21343 ts_tablet_manager.cc:1398] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:37.582365 21340 tablet_bootstrap.cc:492] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:37.582468 21340 ts_tablet_manager.cc:1367] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.002s
I20240502 20:45:37.582988 21340 raft_consensus.cc:348] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:37.583089 21340 raft_consensus.cc:374] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:37.583127 21340 raft_consensus.cc:729] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:37.583197 21338 tablet_bootstrap.cc:654] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:37.583300 21340 consensus_queue.cc:260] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:37.583673 21340 ts_tablet_manager.cc:1398] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:45:37.585458 21337 tablet_bootstrap.cc:492] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:37.585515 21339 tablet_bootstrap.cc:492] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:37.585598 21337 ts_tablet_manager.cc:1367] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 20:45:37.585608 21339 ts_tablet_manager.cc:1367] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.021s	user 0.002s	sys 0.000s
I20240502 20:45:37.586148 21339 raft_consensus.cc:348] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:37.586139 21337 raft_consensus.cc:348] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:37.586279 21337 raft_consensus.cc:374] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:37.586279 21339 raft_consensus.cc:374] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:37.586315 21337 raft_consensus.cc:729] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:37.586328 21339 raft_consensus.cc:729] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:37.586463 21339 consensus_queue.cc:260] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:37.586463 21337 consensus_queue.cc:260] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:37.586779 21339 ts_tablet_manager.cc:1398] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:37.586779 21337 ts_tablet_manager.cc:1398] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:37.587790 21338 tablet_bootstrap.cc:492] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:37.587913 21338 ts_tablet_manager.cc:1367] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 20:45:37.588407 21338 raft_consensus.cc:348] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:37.588604 21338 raft_consensus.cc:374] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:37.588689 21338 raft_consensus.cc:729] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:37.588919 21338 consensus_queue.cc:260] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:37.589370 21338 ts_tablet_manager.cc:1398] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:37.589648 21338 tablet_bootstrap.cc:492] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:37.590961 21338 tablet_bootstrap.cc:654] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:37.592495 21338 tablet_bootstrap.cc:492] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:37.592600 21338 ts_tablet_manager.cc:1367] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:37.594107 21338 raft_consensus.cc:348] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:37.594386 21338 raft_consensus.cc:374] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:37.594604 21338 raft_consensus.cc:729] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:37.594974 21338 consensus_queue.cc:260] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:37.595605 21338 ts_tablet_manager.cc:1398] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:37.679639 21178 raft_consensus.cc:482] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:37.679821 21178 raft_consensus.cc:504] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:37.680367 21178 leader_election.cc:288] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:37.681185 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27c88db38324ae79f3bfe1a91117298" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:37.681350 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27c88db38324ae79f3bfe1a91117298" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:37.681591 19911 raft_consensus.cc:2454] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:37.681594 20451 raft_consensus.cc:2454] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:37.682184 20112 leader_election.cc:302] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 
I20240502 20:45:37.682500 21178 raft_consensus.cc:2781] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:37.682583 21178 raft_consensus.cc:482] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:37.682614 21178 raft_consensus.cc:3037] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.685454 21229 raft_consensus.cc:482] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:37.685602 21229 raft_consensus.cc:504] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:37.686051 21229 leader_election.cc:288] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863), c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391)
I20240502 20:45:37.686600 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8ed69f637141f7b279d8eaced8121f" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:37.686740 20316 raft_consensus.cc:2454] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 0.
I20240502 20:45:37.686790 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8ed69f637141f7b279d8eaced8121f" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8" is_pre_election: true
I20240502 20:45:37.686971 20045 raft_consensus.cc:2454] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 0.
I20240502 20:45:37.687096 20518 leader_election.cc:302] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:37.687319 21229 raft_consensus.cc:2781] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:37.687376 21229 raft_consensus.cc:482] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:37.687404 21229 raft_consensus.cc:3037] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.750316 21157 raft_consensus.cc:482] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:37.750535 21157 raft_consensus.cc:504] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:37.751477 21157 leader_election.cc:288] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:37.752209 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aabafdc9be0486291fbcac1b8e04609" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:37.752209 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aabafdc9be0486291fbcac1b8e04609" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" is_pre_election: true
I20240502 20:45:37.752424 20179 raft_consensus.cc:2454] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:37.752424 20451 raft_consensus.cc:2454] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:37.752964 19843 leader_election.cc:302] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:37.753329 21157 raft_consensus.cc:2781] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:37.753433 21157 raft_consensus.cc:482] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:37.753475 21157 raft_consensus.cc:3037] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.758605 21178 raft_consensus.cc:504] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:37.758626 21229 raft_consensus.cc:504] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:37.758816 21157 raft_consensus.cc:504] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:37.759146 21178 leader_election.cc:288] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Requested vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:37.759527 21157 leader_election.cc:288] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Requested vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:37.759537 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27c88db38324ae79f3bfe1a91117298" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:37.759627 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27c88db38324ae79f3bfe1a91117298" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:37.759680 20451 raft_consensus.cc:3037] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.759735 19911 raft_consensus.cc:3037] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.759994 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aabafdc9be0486291fbcac1b8e04609" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
I20240502 20:45:37.760113 20179 raft_consensus.cc:3037] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.760967 19911 raft_consensus.cc:2454] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:37.761059 20451 raft_consensus.cc:2454] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:37.761122 20179 raft_consensus.cc:2454] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:37.761471 20112 leader_election.cc:302] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 
I20240502 20:45:37.761508 19844 leader_election.cc:302] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:37.761672 21157 raft_consensus.cc:2781] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:37.761849 21157 raft_consensus.cc:686] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [term 1 LEADER]: Becoming Leader. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Running, Role: LEADER
I20240502 20:45:37.762019 21157 consensus_queue.cc:237] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:37.762248 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8ed69f637141f7b279d8eaced8121f" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8"
I20240502 20:45:37.762391 20045 raft_consensus.cc:3037] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.762950 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aabafdc9be0486291fbcac1b8e04609" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:37.763018 21178 raft_consensus.cc:2781] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:37.763098 20451 raft_consensus.cc:3037] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.763108 21178 raft_consensus.cc:686] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 LEADER]: Becoming Leader. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Running, Role: LEADER
I20240502 20:45:37.763262 21178 consensus_queue.cc:237] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:37.763896 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8ed69f637141f7b279d8eaced8121f" candidate_uuid: "fa7ea13bcb524353a03145dda1461c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:37.764026 20316 raft_consensus.cc:3037] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:37.759145 21229 leader_election.cc:288] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 election: Requested vote from peers 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863), c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391)
I20240502 20:45:37.765261 19761 catalog_manager.cc:5484] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131). New cstate: current_term: 1 leader_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:37.765874 19764 catalog_manager.cc:5484] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91942b13253244aab62c2a52e1fdd827 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "91942b13253244aab62c2a52e1fdd827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:37.853827 20316 raft_consensus.cc:2454] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 1.
I20240502 20:45:37.853827 20045 raft_consensus.cc:2454] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7ea13bcb524353a03145dda1461c9e in term 1.
I20240502 20:45:37.853828 20451 raft_consensus.cc:2454] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:37.854432 20518 leader_election.cc:302] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:37.854717 21229 raft_consensus.cc:2781] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:37.854808 21229 raft_consensus.cc:686] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [term 1 LEADER]: Becoming Leader. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Running, Role: LEADER
I20240502 20:45:37.854966 21229 consensus_queue.cc:237] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:37.857893 19764 catalog_manager.cc:5484] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa7ea13bcb524353a03145dda1461c9e (127.19.66.134). New cstate: current_term: 1 leader_uuid: "fa7ea13bcb524353a03145dda1461c9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:38.011883 21358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:38.012300 21358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:38.017053 21358 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 20:45:38.123557 20316 raft_consensus.cc:1261] T ac8ed69f637141f7b279d8eaced8121f P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Refusing update from remote peer fa7ea13bcb524353a03145dda1461c9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.123557 20045 raft_consensus.cc:1261] T ac8ed69f637141f7b279d8eaced8121f P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Refusing update from remote peer fa7ea13bcb524353a03145dda1461c9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.124346 21352 consensus_queue.cc:1035] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.124572 21229 consensus_queue.cc:1035] T ac8ed69f637141f7b279d8eaced8121f P fa7ea13bcb524353a03145dda1461c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.127770 20179 raft_consensus.cc:1261] T 2aabafdc9be0486291fbcac1b8e04609 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.128934 20451 raft_consensus.cc:1261] T 2aabafdc9be0486291fbcac1b8e04609 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.128896 21157 consensus_queue.cc:1035] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.129737 21157 consensus_queue.cc:1035] T 2aabafdc9be0486291fbcac1b8e04609 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.135691 19911 raft_consensus.cc:1261] T b27c88db38324ae79f3bfe1a91117298 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.136170 21178 consensus_queue.cc:1035] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.136799 20450 raft_consensus.cc:1261] T b27c88db38324ae79f3bfe1a91117298 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.138044 21178 consensus_queue.cc:1035] T b27c88db38324ae79f3bfe1a91117298 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.139632 21401 mvcc.cc:205] Tried to move back new op lower bound from 7023340495362449408 to 7023340493876604928. Current Snapshot: MvccSnapshot[applied={T|T < 7023340495362449408}]
I20240502 20:45:38.159235 21394 mvcc.cc:205] Tried to move back new op lower bound from 7023340495383482368 to 7023340493879562240. Current Snapshot: MvccSnapshot[applied={T|T < 7023340495383482368}]
I20240502 20:45:38.163678 21395 mvcc.cc:205] Tried to move back new op lower bound from 7023340495362449408 to 7023340493876604928. Current Snapshot: MvccSnapshot[applied={T|T < 7023340495362449408}]

INSERT report
    rows total: 2000
    time total: 148.007 ms
  time per row: 0.0740034 ms
I20240502 20:45:38.224632 19766 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:38.232780 20565 tablet_service.cc:1467] Processing CreateTablet for tablet 38d37f5cbbac47d2a67fc0f0ba7b62cf (DEFAULT_TABLE table=rebalance_test_table_9 [id=a504101e4f8e418f8a2ecfcf1cbdae39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.233233 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d37f5cbbac47d2a67fc0f0ba7b62cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.233654 20295 tablet_service.cc:1467] Processing CreateTablet for tablet 38d37f5cbbac47d2a67fc0f0ba7b62cf (DEFAULT_TABLE table=rebalance_test_table_9 [id=a504101e4f8e418f8a2ecfcf1cbdae39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.233876 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 3efe7bcb1c0c4923a6b5a29881b0caba (DEFAULT_TABLE table=rebalance_test_table_9 [id=a504101e4f8e418f8a2ecfcf1cbdae39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.233943 20025 tablet_service.cc:1467] Processing CreateTablet for tablet 38d37f5cbbac47d2a67fc0f0ba7b62cf (DEFAULT_TABLE table=rebalance_test_table_9 [id=a504101e4f8e418f8a2ecfcf1cbdae39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.234198 20025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d37f5cbbac47d2a67fc0f0ba7b62cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.234119 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d37f5cbbac47d2a67fc0f0ba7b62cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.234385 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efe7bcb1c0c4923a6b5a29881b0caba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.234978 20026 tablet_service.cc:1467] Processing CreateTablet for tablet 94be83e069b7462aa383f444f95e9d0e (DEFAULT_TABLE table=rebalance_test_table_9 [id=a504101e4f8e418f8a2ecfcf1cbdae39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.235013 20155 tablet_service.cc:1467] Processing CreateTablet for tablet 3efe7bcb1c0c4923a6b5a29881b0caba (DEFAULT_TABLE table=rebalance_test_table_9 [id=a504101e4f8e418f8a2ecfcf1cbdae39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.235312 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efe7bcb1c0c4923a6b5a29881b0caba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.235522 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 3efe7bcb1c0c4923a6b5a29881b0caba (DEFAULT_TABLE table=rebalance_test_table_9 [id=a504101e4f8e418f8a2ecfcf1cbdae39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.235831 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efe7bcb1c0c4923a6b5a29881b0caba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.237504 21427 tablet_bootstrap.cc:492] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:38.237541 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 94be83e069b7462aa383f444f95e9d0e (DEFAULT_TABLE table=rebalance_test_table_9 [id=a504101e4f8e418f8a2ecfcf1cbdae39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.237840 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94be83e069b7462aa383f444f95e9d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.237933 21428 tablet_bootstrap.cc:492] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:38.238561 21427 tablet_bootstrap.cc:654] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.238982 21428 tablet_bootstrap.cc:654] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.240168 21428 tablet_bootstrap.cc:492] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:38.240272 21428 ts_tablet_manager.cc:1367] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:38.240572 20564 tablet_service.cc:1467] Processing CreateTablet for tablet 94be83e069b7462aa383f444f95e9d0e (DEFAULT_TABLE table=rebalance_test_table_9 [id=a504101e4f8e418f8a2ecfcf1cbdae39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.240769 21428 raft_consensus.cc:348] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:38.240900 21428 raft_consensus.cc:374] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.240938 21428 raft_consensus.cc:729] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.241108 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94be83e069b7462aa383f444f95e9d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.241137 21428 consensus_queue.cc:260] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:38.241500 21428 ts_tablet_manager.cc:1398] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:38.241708 21427 tablet_bootstrap.cc:492] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:38.241781 21427 ts_tablet_manager.cc:1367] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:38.242138 21427 raft_consensus.cc:348] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:38.242219 21427 raft_consensus.cc:374] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.242241 21427 raft_consensus.cc:729] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.242367 21427 consensus_queue.cc:260] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:38.242552 21431 tablet_bootstrap.cc:492] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:38.242661 21427 ts_tablet_manager.cc:1398] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:38.243501 21431 tablet_bootstrap.cc:654] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.244541 21427 tablet_bootstrap.cc:492] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:38.244536 20564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94be83e069b7462aa383f444f95e9d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.245709 21427 tablet_bootstrap.cc:654] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.250715 21435 tablet_bootstrap.cc:492] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:38.251716 21435 tablet_bootstrap.cc:654] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.251850 21433 tablet_bootstrap.cc:492] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:38.252931 21426 tablet_bootstrap.cc:492] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:38.252951 21433 tablet_bootstrap.cc:654] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.253297 21435 tablet_bootstrap.cc:492] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:38.253407 21435 ts_tablet_manager.cc:1367] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:38.253990 21426 tablet_bootstrap.cc:654] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.253906 21435 raft_consensus.cc:348] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:38.254036 21435 raft_consensus.cc:374] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.254068 21435 raft_consensus.cc:729] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.254253 21435 consensus_queue.cc:260] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:38.254299 21433 tablet_bootstrap.cc:492] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:38.254408 21433 ts_tablet_manager.cc:1367] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:38.254930 21433 raft_consensus.cc:348] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:38.255033 21433 raft_consensus.cc:374] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.255069 21433 raft_consensus.cc:729] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.255211 21433 consensus_queue.cc:260] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:38.255349 21431 tablet_bootstrap.cc:492] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:38.255474 21431 ts_tablet_manager.cc:1367] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.013s	user 0.002s	sys 0.000s
I20240502 20:45:38.255520 21433 ts_tablet_manager.cc:1398] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:38.255717 21433 tablet_bootstrap.cc:492] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:38.255935 21431 raft_consensus.cc:348] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:38.256045 21431 raft_consensus.cc:374] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.256079 21431 raft_consensus.cc:729] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.256218 21431 consensus_queue.cc:260] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:38.256570 21435 ts_tablet_manager.cc:1398] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:45:38.257315 21433 tablet_bootstrap.cc:654] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.257441 21431 ts_tablet_manager.cc:1398] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:38.257632 21431 tablet_bootstrap.cc:492] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:38.258558 21431 tablet_bootstrap.cc:654] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.258783 21426 tablet_bootstrap.cc:492] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:38.258941 21426 ts_tablet_manager.cc:1367] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 20:45:38.259559 21426 raft_consensus.cc:348] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:38.259723 21426 raft_consensus.cc:374] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.259780 21426 raft_consensus.cc:729] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.259943 21426 consensus_queue.cc:260] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:38.260293 21426 ts_tablet_manager.cc:1398] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:38.261391 21433 tablet_bootstrap.cc:492] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:38.261497 21433 ts_tablet_manager.cc:1367] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 20:45:38.261941 21433 raft_consensus.cc:348] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:38.262079 21433 raft_consensus.cc:374] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.262123 21433 raft_consensus.cc:729] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.262284 21433 consensus_queue.cc:260] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:38.262560 21433 ts_tablet_manager.cc:1398] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:38.265502 21427 tablet_bootstrap.cc:492] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:38.265630 21427 ts_tablet_manager.cc:1367] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.021s	user 0.000s	sys 0.003s
I20240502 20:45:38.266130 21427 raft_consensus.cc:348] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:38.266237 21427 raft_consensus.cc:374] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.266258 21427 raft_consensus.cc:729] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.266377 21427 consensus_queue.cc:260] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:38.266664 21427 ts_tablet_manager.cc:1398] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:38.277637 21431 tablet_bootstrap.cc:492] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:38.277796 21431 ts_tablet_manager.cc:1367] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.000s
I20240502 20:45:38.278291 21431 raft_consensus.cc:348] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:38.278384 21333 raft_consensus.cc:482] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:38.278416 21431 raft_consensus.cc:374] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.278533 21431 raft_consensus.cc:729] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.278546 21333 raft_consensus.cc:504] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:38.278683 21431 consensus_queue.cc:260] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:38.279040 21333 leader_election.cc:288] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393), c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391)
I20240502 20:45:38.279172 21431 ts_tablet_manager.cc:1398] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:38.279541 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d37f5cbbac47d2a67fc0f0ba7b62cf" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:38.279703 20584 raft_consensus.cc:2454] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:38.279835 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d37f5cbbac47d2a67fc0f0ba7b62cf" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:38.279989 20316 raft_consensus.cc:2454] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:38.280041 19981 leader_election.cc:302] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:38.280283 21441 raft_consensus.cc:2781] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:38.280340 21441 raft_consensus.cc:482] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:38.280421 21441 raft_consensus.cc:3037] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.281597 21441 raft_consensus.cc:504] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:38.282079 21441 leader_election.cc:288] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Requested vote from peers fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393), c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391)
I20240502 20:45:38.282500 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d37f5cbbac47d2a67fc0f0ba7b62cf" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:38.282507 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d37f5cbbac47d2a67fc0f0ba7b62cf" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:38.282624 20316 raft_consensus.cc:3037] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.282636 20584 raft_consensus.cc:3037] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.283654 20316 raft_consensus.cc:2454] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:38.283680 20584 raft_consensus.cc:2454] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:38.283962 19979 leader_election.cc:302] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:38.284169 21441 raft_consensus.cc:2781] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.284250 21441 raft_consensus.cc:686] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [term 1 LEADER]: Becoming Leader. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Running, Role: LEADER
I20240502 20:45:38.284394 21441 consensus_queue.cc:237] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } }
I20240502 20:45:38.286137 19761 catalog_manager.cc:5484] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951291b2e23f46a79812e404a06690d8 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "951291b2e23f46a79812e404a06690d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:38.373215 21212 raft_consensus.cc:482] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:38.373409 21212 raft_consensus.cc:504] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:38.373862 21212 leader_election.cc:288] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393), 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:38.374416 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94be83e069b7462aa383f444f95e9d0e" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:38.374433 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94be83e069b7462aa383f444f95e9d0e" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8" is_pre_election: true
I20240502 20:45:38.374567 20584 raft_consensus.cc:2454] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:38.374572 20045 raft_consensus.cc:2454] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:38.374912 20249 leader_election.cc:302] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:38.375102 21212 raft_consensus.cc:2781] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:38.375162 21212 raft_consensus.cc:482] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:38.375214 21212 raft_consensus.cc:3037] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.376149 21212 raft_consensus.cc:504] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:38.376554 21212 leader_election.cc:288] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Requested vote from peers fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393), 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:38.376905 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94be83e069b7462aa383f444f95e9d0e" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:38.377018 20584 raft_consensus.cc:3037] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.376976 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94be83e069b7462aa383f444f95e9d0e" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8"
I20240502 20:45:38.377138 20045 raft_consensus.cc:3037] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.377912 20584 raft_consensus.cc:2454] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:38.377971 20045 raft_consensus.cc:2454] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:38.378299 20249 leader_election.cc:302] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:38.378506 21212 raft_consensus.cc:2781] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.378767 21212 raft_consensus.cc:686] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [term 1 LEADER]: Becoming Leader. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Running, Role: LEADER
I20240502 20:45:38.378943 21212 consensus_queue.cc:237] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:38.381549 19761 catalog_manager.cc:5484] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af reported cstate change: term changed from 0 to 1, leader changed from <none> to c302712f8dbd4fa7bed797758fd772af (127.19.66.132). New cstate: current_term: 1 leader_uuid: "c302712f8dbd4fa7bed797758fd772af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:38.389531 20766 raft_consensus.cc:482] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:38.389729 20766 raft_consensus.cc:504] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:38.390419 20766 leader_election.cc:288] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:38.390980 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe7bcb1c0c4923a6b5a29881b0caba" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:38.390990 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe7bcb1c0c4923a6b5a29881b0caba" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:38.391153 20451 raft_consensus.cc:2454] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:38.391163 19911 raft_consensus.cc:2454] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:38.391479 20112 leader_election.cc:302] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:38.391678 20766 raft_consensus.cc:2781] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:38.391743 20766 raft_consensus.cc:482] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:38.391801 20766 raft_consensus.cc:3037] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.392768 20766 raft_consensus.cc:504] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:38.393178 20766 leader_election.cc:288] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Requested vote from peers 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121), 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:38.393631 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe7bcb1c0c4923a6b5a29881b0caba" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:38.393779 19911 raft_consensus.cc:3037] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.393796 20451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe7bcb1c0c4923a6b5a29881b0caba" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:38.393935 20451 raft_consensus.cc:3037] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.394938 20451 raft_consensus.cc:2454] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:38.394938 19911 raft_consensus.cc:2454] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:38.395326 20112 leader_election.cc:302] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:38.395543 20766 raft_consensus.cc:2781] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.395645 20766 raft_consensus.cc:686] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 LEADER]: Becoming Leader. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Running, Role: LEADER
I20240502 20:45:38.395815 20766 consensus_queue.cc:237] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:38.398542 19761 catalog_manager.cc:5484] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131). New cstate: current_term: 1 leader_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:38.539752 21446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:38.540086 21446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:38.543643 21446 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 20:45:38.650803 20584 raft_consensus.cc:1261] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.650884 20316 raft_consensus.cc:1261] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.651393 21333 consensus_queue.cc:1035] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.651543 21441 consensus_queue.cc:1035] T 38d37f5cbbac47d2a67fc0f0ba7b62cf P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.662063 20045 raft_consensus.cc:1261] T 94be83e069b7462aa383f444f95e9d0e P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Refusing update from remote peer c302712f8dbd4fa7bed797758fd772af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.662565 21443 consensus_queue.cc:1035] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.668895 19911 raft_consensus.cc:1261] T 3efe7bcb1c0c4923a6b5a29881b0caba P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.669114 20451 raft_consensus.cc:1261] T 3efe7bcb1c0c4923a6b5a29881b0caba P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.669661 20766 consensus_queue.cc:1035] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.670035 20766 consensus_queue.cc:1035] T 3efe7bcb1c0c4923a6b5a29881b0caba P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.675035 20584 raft_consensus.cc:1261] T 94be83e069b7462aa383f444f95e9d0e P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Refusing update from remote peer c302712f8dbd4fa7bed797758fd772af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:38.675633 21443 consensus_queue.cc:1035] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:38.711102 21493 mvcc.cc:205] Tried to move back new op lower bound from 7023340497554710528 to 7023340496401641472. Current Snapshot: MvccSnapshot[applied={T|T < 7023340497554710528}]
I20240502 20:45:38.714066 21491 mvcc.cc:205] Tried to move back new op lower bound from 7023340497554710528 to 7023340496401641472. Current Snapshot: MvccSnapshot[applied={T|T < 7023340497554710528}]
I20240502 20:45:38.737201 21396 mvcc.cc:205] Tried to move back new op lower bound from 7023340497554710528 to 7023340496401641472. Current Snapshot: MvccSnapshot[applied={T|T < 7023340497554710528}]

INSERT report
    rows total: 2000
    time total: 155.019 ms
  time per row: 0.0775097 ms
I20240502 20:45:38.757510 19764 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_10"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:38.765247 20431 tablet_service.cc:1467] Processing CreateTablet for tablet c19dad518e984f16989b517b1ce7b377 (DEFAULT_TABLE table=rebalance_test_table_10 [id=a8bbdb29a69f42699db0fb617ecd218d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.765458 19891 tablet_service.cc:1467] Processing CreateTablet for tablet c19dad518e984f16989b517b1ce7b377 (DEFAULT_TABLE table=rebalance_test_table_10 [id=a8bbdb29a69f42699db0fb617ecd218d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.765655 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19dad518e984f16989b517b1ce7b377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.765904 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19dad518e984f16989b517b1ce7b377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.766085 20432 tablet_service.cc:1467] Processing CreateTablet for tablet b1f03c2d324f43b09d994a045413fea2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=a8bbdb29a69f42699db0fb617ecd218d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.766345 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f03c2d324f43b09d994a045413fea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.768216 20026 tablet_service.cc:1467] Processing CreateTablet for tablet 37c622a32f2c444a91476b53d2bee43a (DEFAULT_TABLE table=rebalance_test_table_10 [id=a8bbdb29a69f42699db0fb617ecd218d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.768570 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c622a32f2c444a91476b53d2bee43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.768703 21498 tablet_bootstrap.cc:492] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:38.768872 21499 tablet_bootstrap.cc:492] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:38.769495 20565 tablet_service.cc:1467] Processing CreateTablet for tablet 37c622a32f2c444a91476b53d2bee43a (DEFAULT_TABLE table=rebalance_test_table_10 [id=a8bbdb29a69f42699db0fb617ecd218d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.769819 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c622a32f2c444a91476b53d2bee43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.769850 21498 tablet_bootstrap.cc:654] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.769970 21499 tablet_bootstrap.cc:654] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.770965 21431 tablet_bootstrap.cc:492] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:38.771845 20155 tablet_service.cc:1467] Processing CreateTablet for tablet c19dad518e984f16989b517b1ce7b377 (DEFAULT_TABLE table=rebalance_test_table_10 [id=a8bbdb29a69f42699db0fb617ecd218d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.771956 21431 tablet_bootstrap.cc:654] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.772162 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19dad518e984f16989b517b1ce7b377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.772135 20157 tablet_service.cc:1467] Processing CreateTablet for tablet b1f03c2d324f43b09d994a045413fea2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=a8bbdb29a69f42699db0fb617ecd218d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.772387 20157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f03c2d324f43b09d994a045413fea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.772684 21502 tablet_bootstrap.cc:492] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:38.772840 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 37c622a32f2c444a91476b53d2bee43a (DEFAULT_TABLE table=rebalance_test_table_10 [id=a8bbdb29a69f42699db0fb617ecd218d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.773141 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c622a32f2c444a91476b53d2bee43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.773705 21502 tablet_bootstrap.cc:654] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.774924 21502 tablet_bootstrap.cc:492] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:38.775012 21502 ts_tablet_manager.cc:1367] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:38.775822 21502 raft_consensus.cc:348] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:38.775876 19891 tablet_service.cc:1467] Processing CreateTablet for tablet b1f03c2d324f43b09d994a045413fea2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=a8bbdb29a69f42699db0fb617ecd218d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:38.775951 21502 raft_consensus.cc:374] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.775992 21502 raft_consensus.cc:729] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.776188 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f03c2d324f43b09d994a045413fea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:38.776135 21502 consensus_queue.cc:260] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:38.776458 21502 ts_tablet_manager.cc:1398] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:38.776727 21506 tablet_bootstrap.cc:492] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:38.777813 21506 tablet_bootstrap.cc:654] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.779084 21499 tablet_bootstrap.cc:492] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:38.779198 21499 ts_tablet_manager.cc:1367] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 20:45:38.779368 21498 tablet_bootstrap.cc:492] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:38.779506 21498 ts_tablet_manager.cc:1367] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.000s
I20240502 20:45:38.779711 21499 raft_consensus.cc:348] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:38.779819 21499 raft_consensus.cc:374] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.779863 21499 raft_consensus.cc:729] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.780011 21499 consensus_queue.cc:260] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:38.780099 21498 raft_consensus.cc:348] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:38.780211 21498 raft_consensus.cc:374] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.780247 21498 raft_consensus.cc:729] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.780320 21499 ts_tablet_manager.cc:1398] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:38.780381 21498 consensus_queue.cc:260] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:38.780483 21499 tablet_bootstrap.cc:492] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:38.780673 21498 ts_tablet_manager.cc:1398] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:38.780828 21498 tablet_bootstrap.cc:492] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:38.781829 21498 tablet_bootstrap.cc:654] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.782673 21505 tablet_bootstrap.cc:492] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:38.782752 21499 tablet_bootstrap.cc:654] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.783830 21505 tablet_bootstrap.cc:654] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.783985 21506 tablet_bootstrap.cc:492] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:38.784145 21506 ts_tablet_manager.cc:1367] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 20:45:38.784994 21506 raft_consensus.cc:348] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:38.785238 21506 raft_consensus.cc:374] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.785333 21506 raft_consensus.cc:729] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.785419 21431 tablet_bootstrap.cc:492] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:38.785543 21431 ts_tablet_manager.cc:1367] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.000s
I20240502 20:45:38.785589 21506 consensus_queue.cc:260] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:38.785979 21506 ts_tablet_manager.cc:1398] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:38.786053 21431 raft_consensus.cc:348] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:38.786177 21506 tablet_bootstrap.cc:492] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:38.786176 21431 raft_consensus.cc:374] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.786217 21431 raft_consensus.cc:729] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.786397 21431 consensus_queue.cc:260] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:38.786722 21431 ts_tablet_manager.cc:1398] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:38.788589 21498 tablet_bootstrap.cc:492] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:38.788712 21498 ts_tablet_manager.cc:1367] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 20:45:38.788856 21499 tablet_bootstrap.cc:492] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:38.789016 21499 ts_tablet_manager.cc:1367] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.002s
I20240502 20:45:38.789300 21498 raft_consensus.cc:348] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:38.789443 21498 raft_consensus.cc:374] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.789485 21498 raft_consensus.cc:729] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.789610 21498 consensus_queue.cc:260] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:38.790118 21498 ts_tablet_manager.cc:1398] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:38.790508 21499 raft_consensus.cc:348] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:38.790642 21499 raft_consensus.cc:374] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.790714 21499 raft_consensus.cc:729] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.790870 21499 consensus_queue.cc:260] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:38.791069 21505 tablet_bootstrap.cc:492] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:38.791177 21499 ts_tablet_manager.cc:1398] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 20:45:38.791190 21505 ts_tablet_manager.cc:1367] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.000s
I20240502 20:45:38.791523 21506 tablet_bootstrap.cc:654] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:38.791705 21505 raft_consensus.cc:348] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:38.791821 21505 raft_consensus.cc:374] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.791980 21505 raft_consensus.cc:729] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.792493 21505 consensus_queue.cc:260] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:38.792766 21506 tablet_bootstrap.cc:492] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:38.792868 21506 ts_tablet_manager.cc:1367] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 20:45:38.792883 21505 ts_tablet_manager.cc:1398] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 20:45:38.793411 21506 raft_consensus.cc:348] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:38.793541 21506 raft_consensus.cc:374] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:38.793591 21506 raft_consensus.cc:729] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:38.793711 21506 consensus_queue.cc:260] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:38.794000 21506 ts_tablet_manager.cc:1398] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:38.822252 21484 raft_consensus.cc:482] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:38.822400 21484 raft_consensus.cc:504] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:38.822885 21484 leader_election.cc:288] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:38.823276 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c622a32f2c444a91476b53d2bee43a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:38.823275 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c622a32f2c444a91476b53d2bee43a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:38.823437 20316 raft_consensus.cc:2454] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:38.823437 20584 raft_consensus.cc:2454] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:38.823768 19979 leader_election.cc:302] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:38.823980 21484 raft_consensus.cc:2781] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:38.824048 21484 raft_consensus.cc:482] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:38.824079 21484 raft_consensus.cc:3037] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.825188 21484 raft_consensus.cc:504] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:38.825614 21484 leader_election.cc:288] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Requested vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:38.826100 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c622a32f2c444a91476b53d2bee43a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:38.826135 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c622a32f2c444a91476b53d2bee43a" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:38.826220 20316 raft_consensus.cc:3037] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.826285 20584 raft_consensus.cc:3037] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.827124 20316 raft_consensus.cc:2454] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:38.827148 20584 raft_consensus.cc:2454] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:38.827430 19981 leader_election.cc:302] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:38.827642 21484 raft_consensus.cc:2781] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.827728 21484 raft_consensus.cc:686] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [term 1 LEADER]: Becoming Leader. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Running, Role: LEADER
I20240502 20:45:38.827889 21484 consensus_queue.cc:237] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:38.830617 19760 catalog_manager.cc:5484] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951291b2e23f46a79812e404a06690d8 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "951291b2e23f46a79812e404a06690d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:38.920280 21487 raft_consensus.cc:482] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:38.920432 21487 raft_consensus.cc:504] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:38.920897 21487 leader_election.cc:288] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:38.921356 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19dad518e984f16989b517b1ce7b377" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" is_pre_election: true
I20240502 20:45:38.921527 20179 raft_consensus.cc:2454] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:38.921555 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19dad518e984f16989b517b1ce7b377" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:38.921710 20450 raft_consensus.cc:2454] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:38.921912 19844 leader_election.cc:302] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:38.922135 21487 raft_consensus.cc:2781] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:38.922199 21487 raft_consensus.cc:482] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:38.922251 21487 raft_consensus.cc:3037] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.923300 21487 raft_consensus.cc:504] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:38.923669 21487 leader_election.cc:288] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Requested vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905)
I20240502 20:45:38.924043 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19dad518e984f16989b517b1ce7b377" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:38.924161 20450 raft_consensus.cc:3037] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.924141 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19dad518e984f16989b517b1ce7b377" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
I20240502 20:45:38.924294 20179 raft_consensus.cc:3037] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.925051 20450 raft_consensus.cc:2454] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:38.925226 20179 raft_consensus.cc:2454] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:38.925407 19843 leader_election.cc:302] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:38.925597 21487 raft_consensus.cc:2781] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.925724 21487 raft_consensus.cc:686] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [term 1 LEADER]: Becoming Leader. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Running, Role: LEADER
I20240502 20:45:38.925891 21487 consensus_queue.cc:237] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } }
I20240502 20:45:38.927336 21178 raft_consensus.cc:482] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:38.927462 21178 raft_consensus.cc:504] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:38.927925 21178 leader_election.cc:288] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:38.928556 19760 catalog_manager.cc:5484] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91942b13253244aab62c2a52e1fdd827 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "91942b13253244aab62c2a52e1fdd827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:38.933926 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f03c2d324f43b09d994a045413fea2" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:38.934134 20450 raft_consensus.cc:2454] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:38.934103 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f03c2d324f43b09d994a045413fea2" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827" is_pre_election: true
I20240502 20:45:38.934289 19911 raft_consensus.cc:2454] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 0.
I20240502 20:45:38.934613 20112 leader_election.cc:302] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 
I20240502 20:45:38.934870 21178 raft_consensus.cc:2781] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:38.934998 21178 raft_consensus.cc:482] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:38.935119 21178 raft_consensus.cc:3037] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.936259 21178 raft_consensus.cc:504] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:38.936673 21178 leader_election.cc:288] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Requested vote from peers 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211), 91942b13253244aab62c2a52e1fdd827 (127.19.66.129:37121)
I20240502 20:45:38.937144 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f03c2d324f43b09d994a045413fea2" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:38.937286 20450 raft_consensus.cc:3037] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.937295 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f03c2d324f43b09d994a045413fea2" candidate_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91942b13253244aab62c2a52e1fdd827"
I20240502 20:45:38.937474 19911 raft_consensus.cc:3037] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:38.938256 20450 raft_consensus.cc:2454] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:38.938522 19911 raft_consensus.cc:2454] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900c16672cb34468bbd3e8d3e77ddc1d in term 1.
I20240502 20:45:38.938705 20112 leader_election.cc:302] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5742a696d4c3406086b710e571e2cf0e, 900c16672cb34468bbd3e8d3e77ddc1d; no voters: 
I20240502 20:45:38.939205 21178 raft_consensus.cc:2781] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:38.939321 21178 raft_consensus.cc:686] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 LEADER]: Becoming Leader. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Running, Role: LEADER
I20240502 20:45:38.939527 21178 consensus_queue.cc:237] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } }
I20240502 20:45:38.942593 19760 catalog_manager.cc:5484] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131). New cstate: current_term: 1 leader_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:39.134217 21515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:39.134595 21515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:39.140189 21515 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 20:45:39.278574 20584 raft_consensus.cc:1261] T 37c622a32f2c444a91476b53d2bee43a P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:39.278554 20316 raft_consensus.cc:1261] T 37c622a32f2c444a91476b53d2bee43a P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Refusing update from remote peer 951291b2e23f46a79812e404a06690d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:39.279227 21484 consensus_queue.cc:1035] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:39.279436 21333 consensus_queue.cc:1035] T 37c622a32f2c444a91476b53d2bee43a P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:39.284636 20179 raft_consensus.cc:1261] T c19dad518e984f16989b517b1ce7b377 P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:39.284902 20450 raft_consensus.cc:1261] T c19dad518e984f16989b517b1ce7b377 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:39.285472 21487 consensus_queue.cc:1035] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:39.285662 21404 consensus_queue.cc:1035] T c19dad518e984f16989b517b1ce7b377 P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:39.294184 19911 raft_consensus.cc:1261] T b1f03c2d324f43b09d994a045413fea2 P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:39.295028 21178 consensus_queue.cc:1035] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:39.298177 20451 raft_consensus.cc:1261] T b1f03c2d324f43b09d994a045413fea2 P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 900c16672cb34468bbd3e8d3e77ddc1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:39.298729 21178 consensus_queue.cc:1035] T b1f03c2d324f43b09d994a045413fea2 P 900c16672cb34468bbd3e8d3e77ddc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 146.587 ms
  time per row: 0.0732935 ms
I20240502 20:45:39.374927 19761 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "rebalance_test_table_11"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:39.383612 20565 tablet_service.cc:1467] Processing CreateTablet for tablet fa88a136ba1b4504bd116de3743ab821 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6f9845b515c94b5eb188d670499f40df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.383738 20026 tablet_service.cc:1467] Processing CreateTablet for tablet fa88a136ba1b4504bd116de3743ab821 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6f9845b515c94b5eb188d670499f40df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.384032 20565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa88a136ba1b4504bd116de3743ab821. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.384125 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa88a136ba1b4504bd116de3743ab821. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.384138 20025 tablet_service.cc:1467] Processing CreateTablet for tablet 8566cf4efc9147d89ee0e9f05c87040b (DEFAULT_TABLE table=rebalance_test_table_11 [id=6f9845b515c94b5eb188d670499f40df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.384680 20296 tablet_service.cc:1467] Processing CreateTablet for tablet fa88a136ba1b4504bd116de3743ab821 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6f9845b515c94b5eb188d670499f40df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.384877 20025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8566cf4efc9147d89ee0e9f05c87040b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.384971 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa88a136ba1b4504bd116de3743ab821. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.387079 21588 tablet_bootstrap.cc:492] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:39.387132 20564 tablet_service.cc:1467] Processing CreateTablet for tablet 8566cf4efc9147d89ee0e9f05c87040b (DEFAULT_TABLE table=rebalance_test_table_11 [id=6f9845b515c94b5eb188d670499f40df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.387245 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 99436535192b4bdd9d4fc26f284fc16a (DEFAULT_TABLE table=rebalance_test_table_11 [id=6f9845b515c94b5eb188d670499f40df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.387609 20564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8566cf4efc9147d89ee0e9f05c87040b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.387614 20295 tablet_service.cc:1467] Processing CreateTablet for tablet 8566cf4efc9147d89ee0e9f05c87040b (DEFAULT_TABLE table=rebalance_test_table_11 [id=6f9845b515c94b5eb188d670499f40df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.387737 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99436535192b4bdd9d4fc26f284fc16a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.387876 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8566cf4efc9147d89ee0e9f05c87040b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.388232 21588 tablet_bootstrap.cc:654] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.389990 21587 tablet_bootstrap.cc:492] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:39.391134 20155 tablet_service.cc:1467] Processing CreateTablet for tablet 99436535192b4bdd9d4fc26f284fc16a (DEFAULT_TABLE table=rebalance_test_table_11 [id=6f9845b515c94b5eb188d670499f40df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.391451 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99436535192b4bdd9d4fc26f284fc16a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.391554 21588 tablet_bootstrap.cc:492] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:39.391650 21588 ts_tablet_manager.cc:1367] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:39.391709 21589 tablet_bootstrap.cc:492] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:39.392158 21588 raft_consensus.cc:348] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:39.392362 21588 raft_consensus.cc:374] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.392421 21588 raft_consensus.cc:729] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.392505 21589 tablet_bootstrap.cc:654] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.392572 21588 consensus_queue.cc:260] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:39.392791 21592 tablet_bootstrap.cc:492] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827: Bootstrap starting.
I20240502 20:45:39.392923 21588 ts_tablet_manager.cc:1398] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:39.393338 21588 tablet_bootstrap.cc:492] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e: Bootstrap starting.
I20240502 20:45:39.393703 20432 tablet_service.cc:1467] Processing CreateTablet for tablet 99436535192b4bdd9d4fc26f284fc16a (DEFAULT_TABLE table=rebalance_test_table_11 [id=6f9845b515c94b5eb188d670499f40df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:39.394094 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99436535192b4bdd9d4fc26f284fc16a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:39.394634 21588 tablet_bootstrap.cc:654] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.394703 21589 tablet_bootstrap.cc:492] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:39.394819 21589 ts_tablet_manager.cc:1367] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:45:39.394943 21592 tablet_bootstrap.cc:654] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.395432 21589 raft_consensus.cc:348] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:39.395598 21589 raft_consensus.cc:374] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.395639 21589 raft_consensus.cc:729] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.395810 21589 consensus_queue.cc:260] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:39.396059 21587 tablet_bootstrap.cc:654] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.396160 21589 ts_tablet_manager.cc:1398] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:39.396597 21592 tablet_bootstrap.cc:492] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827: No bootstrap required, opened a new log
I20240502 20:45:39.396761 21592 ts_tablet_manager.cc:1367] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 20:45:39.397722 21592 raft_consensus.cc:348] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:39.397887 21592 raft_consensus.cc:374] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.397924 21592 raft_consensus.cc:729] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.398097 21592 consensus_queue.cc:260] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:39.398458 21592 ts_tablet_manager.cc:1398] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:39.400246 21589 tablet_bootstrap.cc:492] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af: Bootstrap starting.
I20240502 20:45:39.402794 21589 tablet_bootstrap.cc:654] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.404028 21588 tablet_bootstrap.cc:492] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e: No bootstrap required, opened a new log
I20240502 20:45:39.404146 21588 ts_tablet_manager.cc:1367] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.000s
I20240502 20:45:39.404323 21587 tablet_bootstrap.cc:492] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:39.404450 21587 ts_tablet_manager.cc:1367] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.000s
I20240502 20:45:39.404759 21588 raft_consensus.cc:348] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:39.404896 21588 raft_consensus.cc:374] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.404942 21588 raft_consensus.cc:729] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ea13bcb524353a03145dda1461c9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.405107 21588 consensus_queue.cc:260] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:39.405158 21587 raft_consensus.cc:348] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:39.405441 21588 ts_tablet_manager.cc:1398] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:39.405333 21587 raft_consensus.cc:374] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.407625 21587 raft_consensus.cc:729] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.407863 21587 consensus_queue.cc:260] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:39.408418 21587 ts_tablet_manager.cc:1398] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240502 20:45:39.408659 21587 tablet_bootstrap.cc:492] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8: Bootstrap starting.
I20240502 20:45:39.409848 21587 tablet_bootstrap.cc:654] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.411589 21597 tablet_bootstrap.cc:492] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d: Bootstrap starting.
I20240502 20:45:39.412761 21598 tablet_bootstrap.cc:492] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e: Bootstrap starting.
I20240502 20:45:39.413959 21597 tablet_bootstrap.cc:654] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.415059 21598 tablet_bootstrap.cc:654] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:39.416699 21589 tablet_bootstrap.cc:492] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af: No bootstrap required, opened a new log
I20240502 20:45:39.416939 21589 ts_tablet_manager.cc:1367] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af: Time spent bootstrapping tablet: real 0.017s	user 0.002s	sys 0.000s
I20240502 20:45:39.417538 21589 raft_consensus.cc:348] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:39.417701 21589 raft_consensus.cc:374] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.417737 21589 raft_consensus.cc:729] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.417872 21589 consensus_queue.cc:260] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:39.418155 21589 ts_tablet_manager.cc:1398] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:39.419404 21597 tablet_bootstrap.cc:492] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d: No bootstrap required, opened a new log
I20240502 20:45:39.419620 21597 ts_tablet_manager.cc:1367] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 20:45:39.420285 21597 raft_consensus.cc:348] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:39.420521 21597 raft_consensus.cc:374] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.420634 21597 raft_consensus.cc:729] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900c16672cb34468bbd3e8d3e77ddc1d, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.420846 21597 consensus_queue.cc:260] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:39.421273 21597 ts_tablet_manager.cc:1398] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:39.418430 21587 tablet_bootstrap.cc:492] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8: No bootstrap required, opened a new log
I20240502 20:45:39.421689 21587 ts_tablet_manager.cc:1367] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.000s
I20240502 20:45:39.422256 21587 raft_consensus.cc:348] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:39.422392 21587 raft_consensus.cc:374] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.422430 21587 raft_consensus.cc:729] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.422523 21598 tablet_bootstrap.cc:492] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e: No bootstrap required, opened a new log
I20240502 20:45:39.422583 21587 consensus_queue.cc:260] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:39.422765 21598 ts_tablet_manager.cc:1367] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 20:45:39.423368 21598 raft_consensus.cc:348] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:39.423578 21598 raft_consensus.cc:374] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:39.423671 21598 raft_consensus.cc:729] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742a696d4c3406086b710e571e2cf0e, State: Initialized, Role: FOLLOWER
I20240502 20:45:39.424151 21598 consensus_queue.cc:260] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:39.424537 21598 ts_tablet_manager.cc:1398] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:39.427749 21587 ts_tablet_manager.cc:1398] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.000s
I20240502 20:45:39.465626 21333 raft_consensus.cc:482] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:39.465801 21333 raft_consensus.cc:504] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:39.466364 21333 leader_election.cc:288] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:39.466981 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8566cf4efc9147d89ee0e9f05c87040b" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af" is_pre_election: true
I20240502 20:45:39.467007 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8566cf4efc9147d89ee0e9f05c87040b" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:39.467152 20316 raft_consensus.cc:2454] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:39.467192 20584 raft_consensus.cc:2454] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 0.
I20240502 20:45:39.467564 19979 leader_election.cc:302] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:39.467823 21333 raft_consensus.cc:2781] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:39.467893 21333 raft_consensus.cc:482] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:39.467923 21333 raft_consensus.cc:3037] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.468881 21333 raft_consensus.cc:504] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:39.469373 21333 leader_election.cc:288] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Requested vote from peers c302712f8dbd4fa7bed797758fd772af (127.19.66.132:36391), fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393)
I20240502 20:45:39.470078 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8566cf4efc9147d89ee0e9f05c87040b" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:39.470074 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8566cf4efc9147d89ee0e9f05c87040b" candidate_uuid: "951291b2e23f46a79812e404a06690d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c302712f8dbd4fa7bed797758fd772af"
I20240502 20:45:39.470257 20316 raft_consensus.cc:3037] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.470247 20584 raft_consensus.cc:3037] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.471410 20316 raft_consensus.cc:2454] T 8566cf4efc9147d89ee0e9f05c87040b P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:39.471405 20584 raft_consensus.cc:2454] T 8566cf4efc9147d89ee0e9f05c87040b P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951291b2e23f46a79812e404a06690d8 in term 1.
I20240502 20:45:39.472010 19979 leader_election.cc:302] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951291b2e23f46a79812e404a06690d8, c302712f8dbd4fa7bed797758fd772af; no voters: 
I20240502 20:45:39.472369 21333 raft_consensus.cc:2781] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:39.472492 21333 raft_consensus.cc:686] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [term 1 LEADER]: Becoming Leader. State: Replica: 951291b2e23f46a79812e404a06690d8, State: Running, Role: LEADER
I20240502 20:45:39.472687 21333 consensus_queue.cc:237] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } }
I20240502 20:45:39.472961 21212 raft_consensus.cc:482] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:39.473085 21212 raft_consensus.cc:504] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:39.473523 21212 leader_election.cc:288] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393), 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:39.474233 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa88a136ba1b4504bd116de3743ab821" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e" is_pre_election: true
I20240502 20:45:39.474575 20584 raft_consensus.cc:2454] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:39.475018 20249 leader_election.cc:302] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:39.475239 21212 raft_consensus.cc:2781] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:39.475301 21212 raft_consensus.cc:482] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:39.475359 21212 raft_consensus.cc:3037] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.475312 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa88a136ba1b4504bd116de3743ab821" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8" is_pre_election: true
I20240502 20:45:39.475550 20045 raft_consensus.cc:2454] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 0.
I20240502 20:45:39.476495 21212 raft_consensus.cc:504] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:39.476939 21212 leader_election.cc:288] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Requested vote from peers fa7ea13bcb524353a03145dda1461c9e (127.19.66.134:35393), 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863)
I20240502 20:45:39.477761 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa88a136ba1b4504bd116de3743ab821" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951291b2e23f46a79812e404a06690d8"
I20240502 20:45:39.477766 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa88a136ba1b4504bd116de3743ab821" candidate_uuid: "c302712f8dbd4fa7bed797758fd772af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7ea13bcb524353a03145dda1461c9e"
I20240502 20:45:39.477942 20045 raft_consensus.cc:3037] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.477947 20584 raft_consensus.cc:3037] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.478359 19762 catalog_manager.cc:5484] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951291b2e23f46a79812e404a06690d8 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "951291b2e23f46a79812e404a06690d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:39.478984 20045 raft_consensus.cc:2454] T fa88a136ba1b4504bd116de3743ab821 P 951291b2e23f46a79812e404a06690d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:39.479116 20584 raft_consensus.cc:2454] T fa88a136ba1b4504bd116de3743ab821 P fa7ea13bcb524353a03145dda1461c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c302712f8dbd4fa7bed797758fd772af in term 1.
I20240502 20:45:39.479561 20249 leader_election.cc:302] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c302712f8dbd4fa7bed797758fd772af, fa7ea13bcb524353a03145dda1461c9e; no voters: 
I20240502 20:45:39.479786 21212 raft_consensus.cc:2781] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:39.480029 21212 raft_consensus.cc:686] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [term 1 LEADER]: Becoming Leader. State: Replica: c302712f8dbd4fa7bed797758fd772af, State: Running, Role: LEADER
I20240502 20:45:39.480203 21212 consensus_queue.cc:237] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } }
I20240502 20:45:39.482553 19764 catalog_manager.cc:5484] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af reported cstate change: term changed from 0 to 1, leader changed from <none> to c302712f8dbd4fa7bed797758fd772af (127.19.66.132). New cstate: current_term: 1 leader_uuid: "c302712f8dbd4fa7bed797758fd772af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:39.584616 21569 raft_consensus.cc:482] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:39.584805 21569 raft_consensus.cc:504] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:39.585383 21569 leader_election.cc:288] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905), 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:39.585989 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99436535192b4bdd9d4fc26f284fc16a" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" is_pre_election: true
I20240502 20:45:39.586000 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99436535192b4bdd9d4fc26f284fc16a" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e" is_pre_election: true
I20240502 20:45:39.586148 20179 raft_consensus.cc:2454] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:39.586176 20452 raft_consensus.cc:2454] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 0.
I20240502 20:45:39.586512 19844 leader_election.cc:302] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:39.586930 21569 raft_consensus.cc:2781] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:39.587049 21569 raft_consensus.cc:482] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:39.587219 21569 raft_consensus.cc:3037] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.588285 21569 raft_consensus.cc:504] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:39.588766 21569 leader_election.cc:288] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Requested vote from peers 900c16672cb34468bbd3e8d3e77ddc1d (127.19.66.131:44905), 5742a696d4c3406086b710e571e2cf0e (127.19.66.133:34211)
I20240502 20:45:39.589273 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99436535192b4bdd9d4fc26f284fc16a" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5742a696d4c3406086b710e571e2cf0e"
I20240502 20:45:39.589273 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99436535192b4bdd9d4fc26f284fc16a" candidate_uuid: "91942b13253244aab62c2a52e1fdd827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900c16672cb34468bbd3e8d3e77ddc1d"
I20240502 20:45:39.589442 20452 raft_consensus.cc:3037] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.589447 20179 raft_consensus.cc:3037] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:39.590595 20452 raft_consensus.cc:2454] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:39.590595 20179 raft_consensus.cc:2454] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91942b13253244aab62c2a52e1fdd827 in term 1.
I20240502 20:45:39.591048 19844 leader_election.cc:302] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900c16672cb34468bbd3e8d3e77ddc1d, 91942b13253244aab62c2a52e1fdd827; no voters: 
I20240502 20:45:39.591387 21569 raft_consensus.cc:2781] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:39.591495 21569 raft_consensus.cc:686] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [term 1 LEADER]: Becoming Leader. State: Replica: 91942b13253244aab62c2a52e1fdd827, State: Running, Role: LEADER
I20240502 20:45:39.591655 21569 consensus_queue.cc:237] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } }
I20240502 20:45:39.594293 19764 catalog_manager.cc:5484] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91942b13253244aab62c2a52e1fdd827 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "91942b13253244aab62c2a52e1fdd827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91942b13253244aab62c2a52e1fdd827" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 37121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:39.772130 21607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:39.772814 21607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:39.777757 21607 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 20:45:39.862052 21484 consensus_queue.cc:1035] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:39.880016 21333 consensus_queue.cc:1035] T 8566cf4efc9147d89ee0e9f05c87040b P 951291b2e23f46a79812e404a06690d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c302712f8dbd4fa7bed797758fd772af" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:39.880852 21212 consensus_queue.cc:1035] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "951291b2e23f46a79812e404a06690d8" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:39.890267 21604 consensus_queue.cc:1035] T fa88a136ba1b4504bd116de3743ab821 P c302712f8dbd4fa7bed797758fd772af [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7ea13bcb524353a03145dda1461c9e" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:39.907994 20452 raft_consensus.cc:1261] T 99436535192b4bdd9d4fc26f284fc16a P 5742a696d4c3406086b710e571e2cf0e [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:39.908360 20179 raft_consensus.cc:1261] T 99436535192b4bdd9d4fc26f284fc16a P 900c16672cb34468bbd3e8d3e77ddc1d [term 1 FOLLOWER]: Refusing update from remote peer 91942b13253244aab62c2a52e1fdd827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:39.908761 21487 consensus_queue.cc:1035] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5742a696d4c3406086b710e571e2cf0e" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:39.908978 21569 consensus_queue.cc:1035] T 99436535192b4bdd9d4fc26f284fc16a P 91942b13253244aab62c2a52e1fdd827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900c16672cb34468bbd3e8d3e77ddc1d" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 174.296 ms
  time per row: 0.0871478 ms
I20240502 20:45:40.314679 20155 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:40.317276 19891 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:40.324751 20432 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:40.325860 20025 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:40.328259 20295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:40.337906 20564 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:40.493541 20432 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:40.494570 20025 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:40.498358 19891 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:40.502208 20155 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:40.505453 20564 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:40.513370 20295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:40.711997 19891 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:40.714826 20432 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:40.717000 20295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:40.718506 20155 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:40.728804 20564 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:40.731465 20025 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:40.896857 20155 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:40.903775 20564 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:40.903892 20025 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:40.905436 20295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:40.905793 20432 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:40.918658 19891 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:41.081598 20025 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:41.088012 20155 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:41.088486 20295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:41.090873 20432 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:41.097893 20564 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:41.101756 19891 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:41.275600 20432 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:41.275930 20025 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:41.275904 19891 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:45:41.276772 20155 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:45:41.284751 20564 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:41.301952 20295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 62a189e5535d4595bef85381b56a39fb | 127.19.66.190:34537 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                     Value                                                                                     |         Tags          |         Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                           | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                           | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                          | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                             | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                          | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                          | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                          | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                            | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                           | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.19.66.148:37203 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 5742a696d4c3406086b710e571e2cf0e | 127.19.66.133:34211 | HEALTHY | /C       |       4        |       0
 900c16672cb34468bbd3e8d3e77ddc1d | 127.19.66.131:44905 | HEALTHY | /B       |       8        |       0
 91942b13253244aab62c2a52e1fdd827 | 127.19.66.129:37121 | HEALTHY | /A       |       6        |       0
 951291b2e23f46a79812e404a06690d8 | 127.19.66.130:34863 | HEALTHY | /A       |       8        |       0
 c302712f8dbd4fa7bed797758fd772af | 127.19.66.132:36391 | HEALTHY | /B       |       6        |       0
 fa7ea13bcb524353a03145dda1461c9e | 127.19.66.134:35393 | HEALTHY | /C       |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 /A       |       2
 /C       |       2
 /B       |       2

Unusual flags for Tablet Server:
                   Flag                   |                                                                                   Value                                                                                   |         Tags          |      Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                       | experimental          | all 6 server(s) checked
 local_ip_for_outbound_sockets            | 127.19.66.129                                                                                                                                                             | experimental          | 127.19.66.129:37121
 local_ip_for_outbound_sockets            | 127.19.66.130                                                                                                                                                             | experimental          | 127.19.66.130:34863
 local_ip_for_outbound_sockets            | 127.19.66.131                                                                                                                                                             | experimental          | 127.19.66.131:44905
 local_ip_for_outbound_sockets            | 127.19.66.132                                                                                                                                                             | experimental          | 127.19.66.132:36391
 local_ip_for_outbound_sockets            | 127.19.66.133                                                                                                                                                             | experimental          | 127.19.66.133:34211
 local_ip_for_outbound_sockets            | 127.19.66.134                                                                                                                                                             | experimental          | 127.19.66.134:35393
 never_fsync                              | true                                                                                                                                                                      | unsafe,advanced       | all 6 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                         | unsafe,hidden         | all 6 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                      | experimental,advanced | all 6 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                      | unsafe                | all 6 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                        | hidden                | all 6 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.19.66.129:37121
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.19.66.130:34863
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.19.66.131:44905
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.19.66.132:36391
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.19.66.133:34211
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.19.66.134:35393

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.19.66.148:37203 | all 6 server(s) checked
 time_source         | builtin             | all 6 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 7 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name           | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
 rebalance_test_table_0  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_10 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_11 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_5  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_6  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_7  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_8  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_9  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 18
 First Quartile | 18
 Median         | 18
 Third Quartile | 18
 Maximum        | 18

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 6
 Tables         | 12
 Tablets        | 36
 Replicas       | 108

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 20:45:41.335582 19722 log_verifier.cc:126] Checking tablet 07a28fca0dfe49cd907752f80507e547
I20240502 20:45:41.366026 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 07a28fca0dfe49cd907752f80507e547
I20240502 20:45:41.366143 19722 log_verifier.cc:126] Checking tablet 20c70adc4eae4691b41a064fe8d87b68
I20240502 20:45:41.398277 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 20c70adc4eae4691b41a064fe8d87b68
I20240502 20:45:41.398545 19722 log_verifier.cc:126] Checking tablet 2aabafdc9be0486291fbcac1b8e04609
I20240502 20:45:41.432575 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2aabafdc9be0486291fbcac1b8e04609
I20240502 20:45:41.432653 19722 log_verifier.cc:126] Checking tablet 2b69fd1c5a404d2fbe5d3b2f56f09556
I20240502 20:45:41.459666 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2b69fd1c5a404d2fbe5d3b2f56f09556
I20240502 20:45:41.459736 19722 log_verifier.cc:126] Checking tablet 37c622a32f2c444a91476b53d2bee43a
I20240502 20:45:41.488529 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 37c622a32f2c444a91476b53d2bee43a
I20240502 20:45:41.488658 19722 log_verifier.cc:126] Checking tablet 38d37f5cbbac47d2a67fc0f0ba7b62cf
I20240502 20:45:41.517218 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 38d37f5cbbac47d2a67fc0f0ba7b62cf
I20240502 20:45:41.517328 19722 log_verifier.cc:126] Checking tablet 3b08134c7baf48739d2464fc1de7d943
I20240502 20:45:41.546705 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3b08134c7baf48739d2464fc1de7d943
I20240502 20:45:41.546789 19722 log_verifier.cc:126] Checking tablet 3efe7bcb1c0c4923a6b5a29881b0caba
I20240502 20:45:41.579351 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3efe7bcb1c0c4923a6b5a29881b0caba
I20240502 20:45:41.579437 19722 log_verifier.cc:126] Checking tablet 41166a0fb05d4371b6c0c064d169ada4
I20240502 20:45:41.620816 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 41166a0fb05d4371b6c0c064d169ada4
I20240502 20:45:41.620893 19722 log_verifier.cc:126] Checking tablet 4134ba0b3d324d69b69201205a09725d
I20240502 20:45:41.651899 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4134ba0b3d324d69b69201205a09725d
I20240502 20:45:41.652002 19722 log_verifier.cc:126] Checking tablet 478e533f0fe049f7a110b79f5339f8c8
I20240502 20:45:41.682359 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 478e533f0fe049f7a110b79f5339f8c8
I20240502 20:45:41.682448 19722 log_verifier.cc:126] Checking tablet 5a20e3845bcc45b19b0533da1efe1646
I20240502 20:45:41.712289 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5a20e3845bcc45b19b0533da1efe1646
I20240502 20:45:41.712373 19722 log_verifier.cc:126] Checking tablet 5a32010c8c8b42c0a464e4d9f4fd849b
I20240502 20:45:41.746930 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5a32010c8c8b42c0a464e4d9f4fd849b
I20240502 20:45:41.747007 19722 log_verifier.cc:126] Checking tablet 5b1c39c6e15949b3b2fd13a788769a7a
I20240502 20:45:41.776554 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5b1c39c6e15949b3b2fd13a788769a7a
I20240502 20:45:41.776631 19722 log_verifier.cc:126] Checking tablet 768868443e13485398a1c3e23a005083
I20240502 20:45:41.809897 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 768868443e13485398a1c3e23a005083
I20240502 20:45:41.809984 19722 log_verifier.cc:126] Checking tablet 8566cf4efc9147d89ee0e9f05c87040b
I20240502 20:45:41.839969 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8566cf4efc9147d89ee0e9f05c87040b
I20240502 20:45:41.840076 19722 log_verifier.cc:126] Checking tablet 85eb80e5bd9b49109210d7fb915d3e49
I20240502 20:45:41.870154 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 85eb80e5bd9b49109210d7fb915d3e49
I20240502 20:45:41.870410 19722 log_verifier.cc:126] Checking tablet 8be5a401b84340258861b5d5fe02bd09
I20240502 20:45:41.905903 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8be5a401b84340258861b5d5fe02bd09
I20240502 20:45:41.905983 19722 log_verifier.cc:126] Checking tablet 8f5aa53b956b4b129f4cd470f4102f57
I20240502 20:45:41.938484 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8f5aa53b956b4b129f4cd470f4102f57
I20240502 20:45:41.938576 19722 log_verifier.cc:126] Checking tablet 90fd3f27ca0d4e70afc9b2e680455a88
I20240502 20:45:41.972541 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 90fd3f27ca0d4e70afc9b2e680455a88
I20240502 20:45:41.972616 19722 log_verifier.cc:126] Checking tablet 9341fb76068f419b8f2cb05e30cffa9f
I20240502 20:45:42.006637 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9341fb76068f419b8f2cb05e30cffa9f
I20240502 20:45:42.006712 19722 log_verifier.cc:126] Checking tablet 94be83e069b7462aa383f444f95e9d0e
I20240502 20:45:42.039500 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 94be83e069b7462aa383f444f95e9d0e
I20240502 20:45:42.039588 19722 log_verifier.cc:126] Checking tablet 99436535192b4bdd9d4fc26f284fc16a
I20240502 20:45:42.066782 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 99436535192b4bdd9d4fc26f284fc16a
I20240502 20:45:42.066867 19722 log_verifier.cc:126] Checking tablet 9a81c4b4519d4be5a1be4d264fc29c74
I20240502 20:45:42.095976 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9a81c4b4519d4be5a1be4d264fc29c74
I20240502 20:45:42.096063 19722 log_verifier.cc:126] Checking tablet 9f7fda326b754878956cddd3bd5d6037
I20240502 20:45:42.124868 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9f7fda326b754878956cddd3bd5d6037
I20240502 20:45:42.124961 19722 log_verifier.cc:126] Checking tablet ac8ed69f637141f7b279d8eaced8121f
I20240502 20:45:42.154173 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ac8ed69f637141f7b279d8eaced8121f
I20240502 20:45:42.154294 19722 log_verifier.cc:126] Checking tablet b1f03c2d324f43b09d994a045413fea2
I20240502 20:45:42.186811 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b1f03c2d324f43b09d994a045413fea2
I20240502 20:45:42.186897 19722 log_verifier.cc:126] Checking tablet b27c88db38324ae79f3bfe1a91117298
I20240502 20:45:42.217844 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b27c88db38324ae79f3bfe1a91117298
I20240502 20:45:42.217921 19722 log_verifier.cc:126] Checking tablet b56e5eee0c344c03a74f3bb0e62bb8d1
I20240502 20:45:42.249157 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b56e5eee0c344c03a74f3bb0e62bb8d1
I20240502 20:45:42.249235 19722 log_verifier.cc:126] Checking tablet c19dad518e984f16989b517b1ce7b377
I20240502 20:45:42.279932 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c19dad518e984f16989b517b1ce7b377
I20240502 20:45:42.280006 19722 log_verifier.cc:126] Checking tablet d77ddb854c034174babe1a753259cfc8
I20240502 20:45:42.309348 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d77ddb854c034174babe1a753259cfc8
I20240502 20:45:42.309432 19722 log_verifier.cc:126] Checking tablet df575a25f570474f8b67748701025faa
I20240502 20:45:42.342654 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet df575a25f570474f8b67748701025faa
I20240502 20:45:42.342746 19722 log_verifier.cc:126] Checking tablet e4cb9bf04d954ac4a16e71128cbf991a
I20240502 20:45:42.374794 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e4cb9bf04d954ac4a16e71128cbf991a
I20240502 20:45:42.374869 19722 log_verifier.cc:126] Checking tablet e81719d47a2c4d3e8d35ebc674d4fb2a
I20240502 20:45:42.406260 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e81719d47a2c4d3e8d35ebc674d4fb2a
I20240502 20:45:42.406420 19722 log_verifier.cc:126] Checking tablet f8ffa426f4bd4b5fb2c92d59fa4db546
I20240502 20:45:42.441547 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f8ffa426f4bd4b5fb2c92d59fa4db546
I20240502 20:45:42.441620 19722 log_verifier.cc:126] Checking tablet fa88a136ba1b4504bd116de3743ab821
I20240502 20:45:42.475184 19722 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fa88a136ba1b4504bd116de3743ab821
I20240502 20:45:42.495749 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 19827
I20240502 20:45:42.505836 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 19961
W20240502 20:45:42.514897 20249 connection.cc:535] client connection to 127.19.66.130:34863 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:45:42.515094 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 20096
W20240502 20:45:42.515102 20249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:45:42.518010 20249 consensus_peers.cc:485] T 94be83e069b7462aa383f444f95e9d0e P c302712f8dbd4fa7bed797758fd772af -> Peer 951291b2e23f46a79812e404a06690d8 (127.19.66.130:34863): Couldn't send request to peer 951291b2e23f46a79812e404a06690d8. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.130:34863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:42.525233 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 20228
I20240502 20:45:42.534464 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 20369
I20240502 20:45:42.543097 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 20501
I20240502 20:45:42.552927 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 19734
2024-05-02T20:45:42Z chronyd exiting
[       OK ] LocationAwareRebalancingBasicTest.Basic (15513 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (15513 ms total)

[----------] 1 test from RebalanceStartCriteriaTest
[ RUN      ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240502 20:45:42.694977 19722 test_util.cc:268] Using random seed: -1345854405
I20240502 20:45:42.695871 19722 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:42.695920 19722 ts_itest-base.cc:113] --------------
I20240502 20:45:42.695942 19722 ts_itest-base.cc:114] 5 tablet servers
I20240502 20:45:42.695960 19722 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:45:42.695976 19722 ts_itest-base.cc:116] --------------
2024-05-02T20:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:42Z Disabled control of system clock
I20240502 20:45:42.709223 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.190:43875
--webserver_interface=127.19.66.190
--webserver_port=0
--builtin_ntp_servers=127.19.66.148:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.190:43875
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 20:45:42.854912 21973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:42.855633 21973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:42.855825 21973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:42.860285 21973 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:42.860435 21973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:42.860483 21973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:42.860525 21973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:42.860574 21973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:42.866016 21973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.19.66.190:43875
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.190:43875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.66.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.21973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:42.867861 21973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:42.870137 21973 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:42.881451 21978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:42.882166 21979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:42.882476 21981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:42.883769 21973 server_base.cc:1034] running on GCE node
I20240502 20:45:42.884562 21973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:42.885924 21973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:42.887130 21973 hybrid_clock.cc:648] HybridClock initialized: now 1714682742887097 us; error 53 us; skew 500 ppm
I20240502 20:45:42.890722 21973 webserver.cc:460] Webserver started at http://127.19.66.190:40695/ using document root <none> and password file <none>
I20240502 20:45:42.891703 21973 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:42.891798 21973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:42.892087 21973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:42.894726 21973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "19063832c7db41bcab22c3b106f9a522"
format_stamp: "Formatted at 2024-05-02 20:45:42 on dist-test-slave-gqqg"
I20240502 20:45:42.895586 21973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "19063832c7db41bcab22c3b106f9a522"
format_stamp: "Formatted at 2024-05-02 20:45:42 on dist-test-slave-gqqg"
I20240502 20:45:42.901921 21973 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.004s
I20240502 20:45:42.906091 21987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:42.907936 21973 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.003s
I20240502 20:45:42.908126 21973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
uuid: "19063832c7db41bcab22c3b106f9a522"
format_stamp: "Formatted at 2024-05-02 20:45:42 on dist-test-slave-gqqg"
I20240502 20:45:42.908293 21973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:42.939319 21973 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:42.940133 21973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:42.940312 21973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:42.948169 21973 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.190:43875
I20240502 20:45:42.948233 22040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.190:43875 every 8 connection(s)
I20240502 20:45:42.949437 21973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:45:42.952662 22041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:42.958833 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 21973
I20240502 20:45:42.958962 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:45:42.959051 22041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522: Bootstrap starting.
I20240502 20:45:42.961839 22041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:42.962631 22041 log.cc:824] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:42.964511 22041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522: No bootstrap required, opened a new log
I20240502 20:45:42.968371 22041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19063832c7db41bcab22c3b106f9a522" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43875 } }
I20240502 20:45:42.968786 22041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:42.968894 22041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19063832c7db41bcab22c3b106f9a522, State: Initialized, Role: FOLLOWER
I20240502 20:45:42.969550 22041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19063832c7db41bcab22c3b106f9a522" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43875 } }
I20240502 20:45:42.969740 22041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:42.969798 22041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:42.969928 22041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:42.971064 22041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19063832c7db41bcab22c3b106f9a522" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43875 } }
I20240502 20:45:42.971513 22041 leader_election.cc:302] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19063832c7db41bcab22c3b106f9a522; no voters: 
I20240502 20:45:42.971940 22041 leader_election.cc:288] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 20:45:42.972194 22046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:42.972575 22046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [term 1 LEADER]: Becoming Leader. State: Replica: 19063832c7db41bcab22c3b106f9a522, State: Running, Role: LEADER
I20240502 20:45:42.973024 22046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19063832c7db41bcab22c3b106f9a522" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43875 } }
I20240502 20:45:42.973649 22041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:42.975337 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19063832c7db41bcab22c3b106f9a522. Latest consensus state: current_term: 1 leader_uuid: "19063832c7db41bcab22c3b106f9a522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19063832c7db41bcab22c3b106f9a522" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43875 } } }
I20240502 20:45:42.975646 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:42.976090 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19063832c7db41bcab22c3b106f9a522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19063832c7db41bcab22c3b106f9a522" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43875 } } }
I20240502 20:45:42.976224 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:42.976393 22054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:42.979844 22054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:42.986500 22054 catalog_manager.cc:1347] Generated new cluster ID: a1bfef3b62214cefa122bfd27731adbf
I20240502 20:45:42.986617 22054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:42.998646 22054 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:42.999814 22054 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:43.007803 22054 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 19063832c7db41bcab22c3b106f9a522: Generated new TSK 0
I20240502 20:45:43.008724 22054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:43.014942 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.129:0
--local_ip_for_outbound_sockets=127.19.66.129
--webserver_interface=127.19.66.129
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43875
--builtin_ntp_servers=127.19.66.148:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:43.158847 22065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:43.159173 22065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:43.159279 22065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:43.163429 22065 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:43.163553 22065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:43.163678 22065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.129
I20240502 20:45:43.168123 22065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.66.129
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43875
--never_fsync=true
--heap_profile_path=/tmp/kudu.22065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.129
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:43.169457 22065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:43.171041 22065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:43.184607 22070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.185206 22073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.186067 22071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:43.186766 22065 server_base.cc:1034] running on GCE node
I20240502 20:45:43.187420 22065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:43.188175 22065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:43.189392 22065 hybrid_clock.cc:648] HybridClock initialized: now 1714682743189350 us; error 65 us; skew 500 ppm
I20240502 20:45:43.192225 22065 webserver.cc:460] Webserver started at http://127.19.66.129:39887/ using document root <none> and password file <none>
I20240502 20:45:43.193801 22065 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:43.193882 22065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:43.194123 22065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:43.196535 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fd9e3743a8b44cd58c819bd985de791e"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.197196 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fd9e3743a8b44cd58c819bd985de791e"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.201562 22065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.002s
I20240502 20:45:43.204560 22079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.205993 22065 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:43.206164 22065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fd9e3743a8b44cd58c819bd985de791e"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.206295 22065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:43.232148 22065 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:43.233033 22065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:43.233296 22065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:43.233985 22065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:43.235183 22065 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:43.235237 22065 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.235306 22065 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:43.235339 22065 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.247879 22065 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.129:41063
I20240502 20:45:43.247893 22196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.129:41063 every 8 connection(s)
I20240502 20:45:43.268465 22065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:45:43.275673 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 22065
I20240502 20:45:43.275820 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:45:43.278489 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.130:0
--local_ip_for_outbound_sockets=127.19.66.130
--webserver_interface=127.19.66.130
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43875
--builtin_ntp_servers=127.19.66.148:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:43.280854 22197 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43875
I20240502 20:45:43.281183 22197 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:43.281827 22197 heartbeater.cc:511] Master 127.19.66.190:43875 requested a full tablet report, sending...
I20240502 20:45:43.283860 22003 ts_manager.cc:194] Registered new tserver with Master: fd9e3743a8b44cd58c819bd985de791e (127.19.66.129:41063)
I20240502 20:45:43.285910 22003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.129:38113
W20240502 20:45:43.431452 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:43.431742 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:43.431794 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:43.435288 22200 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:43.435391 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:43.435470 22200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.130
I20240502 20:45:43.439525 22200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.66.130
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43875
--never_fsync=true
--heap_profile_path=/tmp/kudu.22200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.130
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:43.440611 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:43.441902 22200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:43.452176 22208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.452744 22205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.453253 22206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:43.454247 22200 server_base.cc:1034] running on GCE node
I20240502 20:45:43.455027 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:43.455780 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:43.456972 22200 hybrid_clock.cc:648] HybridClock initialized: now 1714682743456908 us; error 90 us; skew 500 ppm
I20240502 20:45:43.460625 22200 webserver.cc:460] Webserver started at http://127.19.66.130:33129/ using document root <none> and password file <none>
I20240502 20:45:43.461479 22200 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:43.461563 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:43.461791 22200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:43.463884 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f91715a27e0649e2a963964eb8816816"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.464483 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f91715a27e0649e2a963964eb8816816"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.468955 22200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.002s
I20240502 20:45:43.472213 22215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.473649 22200 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:43.473789 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f91715a27e0649e2a963964eb8816816"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.473901 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:43.501397 22200 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:43.502076 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:43.502238 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:43.502815 22200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:43.503808 22200 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:43.503861 22200 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.503906 22200 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:43.503921 22200 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.515486 22200 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.130:46535
I20240502 20:45:43.515522 22329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.130:46535 every 8 connection(s)
I20240502 20:45:43.516667 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:45:43.517601 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 22200
I20240502 20:45:43.517711 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:45:43.520900 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.131:0
--local_ip_for_outbound_sockets=127.19.66.131
--webserver_interface=127.19.66.131
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43875
--builtin_ntp_servers=127.19.66.148:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:43.529461 22330 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43875
I20240502 20:45:43.529750 22330 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:43.530699 22330 heartbeater.cc:511] Master 127.19.66.190:43875 requested a full tablet report, sending...
I20240502 20:45:43.532092 22003 ts_manager.cc:194] Registered new tserver with Master: f91715a27e0649e2a963964eb8816816 (127.19.66.130:46535)
I20240502 20:45:43.532742 22003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.130:49299
W20240502 20:45:43.652209 22334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:43.652554 22334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:43.652642 22334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:43.656458 22334 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:43.656574 22334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:43.656698 22334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.131
I20240502 20:45:43.660987 22334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.66.131
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43875
--never_fsync=true
--heap_profile_path=/tmp/kudu.22334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.131
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:43.662204 22334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:43.663493 22334 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:43.673444 22342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.674031 22340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.674855 22339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:43.675729 22334 server_base.cc:1034] running on GCE node
I20240502 20:45:43.676357 22334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:43.677134 22334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:43.678344 22334 hybrid_clock.cc:648] HybridClock initialized: now 1714682743678317 us; error 44 us; skew 500 ppm
I20240502 20:45:43.680976 22334 webserver.cc:460] Webserver started at http://127.19.66.131:39751/ using document root <none> and password file <none>
I20240502 20:45:43.681712 22334 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:43.681783 22334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:43.681986 22334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:43.683920 22334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5960bff2ecf14b2db188278403149b9e"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.684473 22334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5960bff2ecf14b2db188278403149b9e"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.688442 22334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20240502 20:45:43.691251 22348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.692508 22334 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:43.692652 22334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5960bff2ecf14b2db188278403149b9e"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.692764 22334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:43.715179 22334 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:43.715970 22334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:43.716179 22334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:43.716897 22334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:43.718128 22334 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:43.718184 22334 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.718252 22334 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:43.718410 22334 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.729157 22334 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.131:33425
I20240502 20:45:43.729321 22466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.131:33425 every 8 connection(s)
I20240502 20:45:43.730336 22334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:45:43.740180 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 22334
I20240502 20:45:43.740339 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:45:43.740550 22467 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43875
I20240502 20:45:43.740844 22467 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:43.741585 22467 heartbeater.cc:511] Master 127.19.66.190:43875 requested a full tablet report, sending...
I20240502 20:45:43.742846 22003 ts_manager.cc:194] Registered new tserver with Master: 5960bff2ecf14b2db188278403149b9e (127.19.66.131:33425)
I20240502 20:45:43.743151 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.132:0
--local_ip_for_outbound_sockets=127.19.66.132
--webserver_interface=127.19.66.132
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43875
--builtin_ntp_servers=127.19.66.148:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:43.743520 22003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.131:46671
W20240502 20:45:43.873414 22470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:43.873725 22470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:43.873781 22470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:43.877383 22470 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:43.877493 22470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:43.877575 22470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.132
I20240502 20:45:43.882220 22470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.66.132
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43875
--never_fsync=true
--heap_profile_path=/tmp/kudu.22470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.132
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:43.883364 22470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:43.884727 22470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:43.895680 22475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.896073 22476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:43.896696 22478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:43.897434 22470 server_base.cc:1034] running on GCE node
I20240502 20:45:43.898485 22470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:43.899318 22470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:43.900501 22470 hybrid_clock.cc:648] HybridClock initialized: now 1714682743900473 us; error 49 us; skew 500 ppm
I20240502 20:45:43.903096 22470 webserver.cc:460] Webserver started at http://127.19.66.132:38009/ using document root <none> and password file <none>
I20240502 20:45:43.903807 22470 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:43.903882 22470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:43.904089 22470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:43.906227 22470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "82142d93671c41daa192655de1c8dc66"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.906821 22470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "82142d93671c41daa192655de1c8dc66"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.911396 22470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 20:45:43.914358 22484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.915627 22470 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:43.915768 22470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "82142d93671c41daa192655de1c8dc66"
format_stamp: "Formatted at 2024-05-02 20:45:43 on dist-test-slave-gqqg"
I20240502 20:45:43.915897 22470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:43.937685 22470 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:43.938452 22470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:43.938644 22470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:43.939430 22470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:43.940505 22470 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:43.940567 22470 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.940640 22470 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:43.940668 22470 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:43.950824 22470 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.132:44441
I20240502 20:45:43.950863 22600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.132:44441 every 8 connection(s)
I20240502 20:45:43.951987 22470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:43.961621 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 22470
I20240502 20:45:43.961776 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:45:43.964339 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.133:0
--local_ip_for_outbound_sockets=127.19.66.133
--webserver_interface=127.19.66.133
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43875
--builtin_ntp_servers=127.19.66.148:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:43.964820 22601 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43875
I20240502 20:45:43.965137 22601 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:43.965814 22601 heartbeater.cc:511] Master 127.19.66.190:43875 requested a full tablet report, sending...
I20240502 20:45:43.967018 22003 ts_manager.cc:194] Registered new tserver with Master: 82142d93671c41daa192655de1c8dc66 (127.19.66.132:44441)
I20240502 20:45:43.967650 22003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.132:41963
W20240502 20:45:44.089651 22604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:44.089941 22604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:44.090130 22604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:44.093819 22604 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:44.093922 22604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:44.094033 22604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.133
I20240502 20:45:44.099009 22604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.66.133
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43875
--never_fsync=true
--heap_profile_path=/tmp/kudu.22604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.133
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:44.100497 22604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:44.101979 22604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:44.112068 22612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:44.112721 22604 server_base.cc:1034] running on GCE node
W20240502 20:45:44.113361 22609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:44.113973 22610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:44.114522 22604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:44.115278 22604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:44.116468 22604 hybrid_clock.cc:648] HybridClock initialized: now 1714682744116449 us; error 38 us; skew 500 ppm
I20240502 20:45:44.119622 22604 webserver.cc:460] Webserver started at http://127.19.66.133:38459/ using document root <none> and password file <none>
I20240502 20:45:44.120710 22604 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:44.120832 22604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:44.121171 22604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:44.123323 22604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "705889764a734a37b5a1f418a54655c8"
format_stamp: "Formatted at 2024-05-02 20:45:44 on dist-test-slave-gqqg"
I20240502 20:45:44.123936 22604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "705889764a734a37b5a1f418a54655c8"
format_stamp: "Formatted at 2024-05-02 20:45:44 on dist-test-slave-gqqg"
I20240502 20:45:44.128377 22604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240502 20:45:44.131196 22618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:44.132433 22604 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:45:44.132575 22604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "705889764a734a37b5a1f418a54655c8"
format_stamp: "Formatted at 2024-05-02 20:45:44 on dist-test-slave-gqqg"
I20240502 20:45:44.132704 22604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:44.154958 22604 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:44.155810 22604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:44.156025 22604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:44.156731 22604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:44.157871 22604 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:44.157931 22604 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:44.158001 22604 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:44.158038 22604 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:44.169001 22604 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.133:45567
I20240502 20:45:44.169035 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.133:45567 every 8 connection(s)
I20240502 20:45:44.170328 22604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:44.172808 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 22604
I20240502 20:45:44.172948 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:45:44.186988 22737 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43875
I20240502 20:45:44.187306 22737 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:44.188129 22737 heartbeater.cc:511] Master 127.19.66.190:43875 requested a full tablet report, sending...
I20240502 20:45:44.189774 22003 ts_manager.cc:194] Registered new tserver with Master: 705889764a734a37b5a1f418a54655c8 (127.19.66.133:45567)
I20240502 20:45:44.190711 22003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.133:37523
I20240502 20:45:44.200893 19722 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 20:45:44.212792 22003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37898:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 20:45:44.240032 22398 tablet_service.cc:1467] Processing CreateTablet for tablet be1df3f588854952b46276d4f8ac3a17 (DEFAULT_TABLE table=TestTable [id=a0d3a99973f04bcf9cd00360163977c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:44.239986 22129 tablet_service.cc:1467] Processing CreateTablet for tablet be1df3f588854952b46276d4f8ac3a17 (DEFAULT_TABLE table=TestTable [id=a0d3a99973f04bcf9cd00360163977c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:44.241125 22398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1df3f588854952b46276d4f8ac3a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.241182 22129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1df3f588854952b46276d4f8ac3a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.241123 22534 tablet_service.cc:1467] Processing CreateTablet for tablet be1df3f588854952b46276d4f8ac3a17 (DEFAULT_TABLE table=TestTable [id=a0d3a99973f04bcf9cd00360163977c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:44.242149 22534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1df3f588854952b46276d4f8ac3a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:44.247018 22756 tablet_bootstrap.cc:492] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e: Bootstrap starting.
I20240502 20:45:44.249572 22756 tablet_bootstrap.cc:654] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:44.250627 22756 log.cc:824] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:44.251397 22758 tablet_bootstrap.cc:492] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66: Bootstrap starting.
I20240502 20:45:44.251632 22757 tablet_bootstrap.cc:492] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e: Bootstrap starting.
I20240502 20:45:44.253641 22756 tablet_bootstrap.cc:492] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e: No bootstrap required, opened a new log
I20240502 20:45:44.253865 22756 ts_tablet_manager.cc:1367] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 20:45:44.253877 22758 tablet_bootstrap.cc:654] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:44.254523 22757 tablet_bootstrap.cc:654] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:44.255270 22757 log.cc:824] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:44.256001 22758 log.cc:824] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:44.257251 22757 tablet_bootstrap.cc:492] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e: No bootstrap required, opened a new log
I20240502 20:45:44.257467 22757 ts_tablet_manager.cc:1367] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 20:45:44.257431 22756 raft_consensus.cc:348] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 } } peers { permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 } } peers { permanent_uuid: "82142d93671c41daa192655de1c8dc66" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 44441 } }
I20240502 20:45:44.257699 22756 raft_consensus.cc:374] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:44.257762 22756 raft_consensus.cc:729] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5960bff2ecf14b2db188278403149b9e, State: Initialized, Role: FOLLOWER
I20240502 20:45:44.257942 22758 tablet_bootstrap.cc:492] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66: No bootstrap required, opened a new log
I20240502 20:45:44.258129 22758 ts_tablet_manager.cc:1367] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 20:45:44.258270 22756 consensus_queue.cc:260] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 } } peers { permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 } } peers { permanent_uuid: "82142d93671c41daa192655de1c8dc66" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 44441 } }
I20240502 20:45:44.259323 22756 ts_tablet_manager.cc:1398] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 20:45:44.259346 22467 heartbeater.cc:503] Master 127.19.66.190:43875 was elected leader, sending a full tablet report...
I20240502 20:45:44.261981 22757 raft_consensus.cc:348] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 } } peers { permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 } } peers { permanent_uuid: "82142d93671c41daa192655de1c8dc66" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 44441 } }
I20240502 20:45:44.262256 22757 raft_consensus.cc:374] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:44.262328 22757 raft_consensus.cc:729] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9e3743a8b44cd58c819bd985de791e, State: Initialized, Role: FOLLOWER
I20240502 20:45:44.262656 22758 raft_consensus.cc:348] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 } } peers { permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 } } peers { permanent_uuid: "82142d93671c41daa192655de1c8dc66" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 44441 } }
I20240502 20:45:44.262872 22758 raft_consensus.cc:374] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:44.262918 22758 raft_consensus.cc:729] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82142d93671c41daa192655de1c8dc66, State: Initialized, Role: FOLLOWER
I20240502 20:45:44.262843 22757 consensus_queue.cc:260] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 } } peers { permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 } } peers { permanent_uuid: "82142d93671c41daa192655de1c8dc66" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 44441 } }
I20240502 20:45:44.263408 22758 consensus_queue.cc:260] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 } } peers { permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 } } peers { permanent_uuid: "82142d93671c41daa192655de1c8dc66" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 44441 } }
I20240502 20:45:44.263761 22197 heartbeater.cc:503] Master 127.19.66.190:43875 was elected leader, sending a full tablet report...
I20240502 20:45:44.263940 22757 ts_tablet_manager.cc:1398] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240502 20:45:44.264287 22758 ts_tablet_manager.cc:1398] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240502 20:45:44.264770 22601 heartbeater.cc:503] Master 127.19.66.190:43875 was elected leader, sending a full tablet report...
W20240502 20:45:44.270474 22198 tablet.cc:2369] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:44.437165 22764 raft_consensus.cc:482] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:44.437418 22764 raft_consensus.cc:504] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 } } peers { permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 } } peers { permanent_uuid: "82142d93671c41daa192655de1c8dc66" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 44441 } }
I20240502 20:45:44.438732 22764 leader_election.cc:288] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd9e3743a8b44cd58c819bd985de791e (127.19.66.129:41063), 5960bff2ecf14b2db188278403149b9e (127.19.66.131:33425)
I20240502 20:45:44.442646 22419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1df3f588854952b46276d4f8ac3a17" candidate_uuid: "82142d93671c41daa192655de1c8dc66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5960bff2ecf14b2db188278403149b9e" is_pre_election: true
I20240502 20:45:44.442646 22150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1df3f588854952b46276d4f8ac3a17" candidate_uuid: "82142d93671c41daa192655de1c8dc66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9e3743a8b44cd58c819bd985de791e" is_pre_election: true
I20240502 20:45:44.442947 22150 raft_consensus.cc:2454] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82142d93671c41daa192655de1c8dc66 in term 0.
I20240502 20:45:44.442948 22419 raft_consensus.cc:2454] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82142d93671c41daa192655de1c8dc66 in term 0.
I20240502 20:45:44.443437 22488 leader_election.cc:302] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5960bff2ecf14b2db188278403149b9e, 82142d93671c41daa192655de1c8dc66; no voters: 
I20240502 20:45:44.443706 22764 raft_consensus.cc:2781] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:44.443782 22764 raft_consensus.cc:482] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:44.443823 22764 raft_consensus.cc:3037] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:44.444768 22764 raft_consensus.cc:504] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 } } peers { permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 } } peers { permanent_uuid: "82142d93671c41daa192655de1c8dc66" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 44441 } }
I20240502 20:45:44.445178 22764 leader_election.cc:288] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [CANDIDATE]: Term 1 election: Requested vote from peers fd9e3743a8b44cd58c819bd985de791e (127.19.66.129:41063), 5960bff2ecf14b2db188278403149b9e (127.19.66.131:33425)
I20240502 20:45:44.445631 22419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1df3f588854952b46276d4f8ac3a17" candidate_uuid: "82142d93671c41daa192655de1c8dc66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5960bff2ecf14b2db188278403149b9e"
I20240502 20:45:44.445680 22150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1df3f588854952b46276d4f8ac3a17" candidate_uuid: "82142d93671c41daa192655de1c8dc66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9e3743a8b44cd58c819bd985de791e"
I20240502 20:45:44.445793 22419 raft_consensus.cc:3037] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:44.445806 22150 raft_consensus.cc:3037] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:44.447151 22150 raft_consensus.cc:2454] T be1df3f588854952b46276d4f8ac3a17 P fd9e3743a8b44cd58c819bd985de791e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82142d93671c41daa192655de1c8dc66 in term 1.
I20240502 20:45:44.447152 22419 raft_consensus.cc:2454] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82142d93671c41daa192655de1c8dc66 in term 1.
I20240502 20:45:44.447525 22488 leader_election.cc:302] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5960bff2ecf14b2db188278403149b9e, 82142d93671c41daa192655de1c8dc66; no voters: 
I20240502 20:45:44.447705 22764 raft_consensus.cc:2781] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:44.447945 22764 raft_consensus.cc:686] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [term 1 LEADER]: Becoming Leader. State: Replica: 82142d93671c41daa192655de1c8dc66, State: Running, Role: LEADER
I20240502 20:45:44.448300 22764 consensus_queue.cc:237] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 } } peers { permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 } } peers { permanent_uuid: "82142d93671c41daa192655de1c8dc66" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 44441 } }
I20240502 20:45:44.451850 22003 catalog_manager.cc:5484] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82142d93671c41daa192655de1c8dc66 (127.19.66.132). New cstate: current_term: 1 leader_uuid: "82142d93671c41daa192655de1c8dc66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82142d93671c41daa192655de1c8dc66" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 44441 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:44.453271 22602 tablet.cc:2369] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:44.481491 19722 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
W20240502 20:45:44.481768 22468 tablet.cc:2369] T be1df3f588854952b46276d4f8ac3a17 P 5960bff2ecf14b2db188278403149b9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:44.483672 19722 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fd9e3743a8b44cd58c819bd985de791e to finish bootstrapping
I20240502 20:45:44.490725 19722 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f91715a27e0649e2a963964eb8816816 to finish bootstrapping
I20240502 20:45:44.498488 19722 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5960bff2ecf14b2db188278403149b9e to finish bootstrapping
I20240502 20:45:44.504997 19722 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 82142d93671c41daa192655de1c8dc66 to finish bootstrapping
I20240502 20:45:44.512131 19722 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 705889764a734a37b5a1f418a54655c8 to finish bootstrapping
I20240502 20:45:44.535636 22330 heartbeater.cc:503] Master 127.19.66.190:43875 was elected leader, sending a full tablet report...
I20240502 20:45:44.769991 22398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:44.770913 22264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:44.771036 22534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:44.772554 22666 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:44.774132 22129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:44.911273 22534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:44.911370 22264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:44.912120 22666 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:44.912912 22129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:44.913894 22398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:44.968626 22840 consensus_queue.cc:1035] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5960bff2ecf14b2db188278403149b9e" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 33425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:44.980172 22846 consensus_queue.cc:1035] T be1df3f588854952b46276d4f8ac3a17 P 82142d93671c41daa192655de1c8dc66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9e3743a8b44cd58c819bd985de791e" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 41063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:45.193413 22737 heartbeater.cc:503] Master 127.19.66.190:43875 was elected leader, sending a full tablet report...
I20240502 20:45:45.207677 22534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:45.210144 22129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.210268 22398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.211571 22264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.212585 22666 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.333039 22264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.333107 22666 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.333106 22129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.335510 22398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.336038 22534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:45.452581 22129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.452804 22264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.452914 22398 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.453936 22534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:45:45.457376 22666 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:45.502106 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 22065
I20240502 20:45:45.509850 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 22200
I20240502 20:45:45.517534 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 22334
I20240502 20:45:45.525358 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 22470
I20240502 20:45:45.532918 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 22604
I20240502 20:45:45.540091 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 21973
2024-05-02T20:45:45Z chronyd exiting
[       OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (2871 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2871 ms total)

[----------] 1 test from RebalanceParamTest
[ RUN      ] RebalanceParamTest.Rebalance/3
I20240502 20:45:45.566299 19722 test_util.cc:268] Using random seed: -1342983082
I20240502 20:45:45.567167 19722 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:45:45.567214 19722 ts_itest-base.cc:113] --------------
I20240502 20:45:45.567224 19722 ts_itest-base.cc:114] 7 tablet servers
I20240502 20:45:45.567235 19722 ts_itest-base.cc:115] 2 replicas per TS
I20240502 20:45:45.567246 19722 ts_itest-base.cc:116] --------------
2024-05-02T20:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:45:45Z Disabled control of system clock
I20240502 20:45:45.578822 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.190:46265
--webserver_interface=127.19.66.190
--webserver_port=0
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.190:46265
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 20:45:45.719135 22956 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 20:45:45.719645 22956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:45.719712 22956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:45.719800 22956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:45.725178 22956 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:45.725348 22956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:45:45.725411 22956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:45.725452 22956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:45:45.725503 22956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:45:45.730435 22956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.19.66.190:46265
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.190:46265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.66.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.22956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:45.731647 22956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:45.732993 22956 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:45.742756 22961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:45.743240 22964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:45.743566 22962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:45.744335 22956 server_base.cc:1034] running on GCE node
I20240502 20:45:45.745030 22956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:45.746206 22956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:45.747413 22956 hybrid_clock.cc:648] HybridClock initialized: now 1714682745747399 us; error 36 us; skew 500 ppm
I20240502 20:45:45.749948 22956 webserver.cc:460] Webserver started at http://127.19.66.190:46045/ using document root <none> and password file <none>
I20240502 20:45:45.750631 22956 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:45.750706 22956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:45.750911 22956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:45.752959 22956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2aa045eae84748a59afc512c822816b6"
format_stamp: "Formatted at 2024-05-02 20:45:45 on dist-test-slave-gqqg"
I20240502 20:45:45.753633 22956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2aa045eae84748a59afc512c822816b6"
format_stamp: "Formatted at 2024-05-02 20:45:45 on dist-test-slave-gqqg"
I20240502 20:45:45.758033 22956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 20:45:45.761577 22970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:45.762931 22956 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:45.763110 22956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2aa045eae84748a59afc512c822816b6"
format_stamp: "Formatted at 2024-05-02 20:45:45 on dist-test-slave-gqqg"
I20240502 20:45:45.763235 22956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:45.781668 22956 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:45.782502 22956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:45.782697 22956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:45.791015 22956 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.190:46265
I20240502 20:45:45.791067 23023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.190:46265 every 8 connection(s)
I20240502 20:45:45.792171 22956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:45:45.795219 23024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:45.798207 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 22956
I20240502 20:45:45.798357 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:45:45.801884 23024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6: Bootstrap starting.
I20240502 20:45:45.804476 23024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:45.805549 23024 log.cc:824] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:45.808146 23024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6: No bootstrap required, opened a new log
I20240502 20:45:45.811223 23024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa045eae84748a59afc512c822816b6" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 46265 } }
I20240502 20:45:45.811476 23024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:45.811514 23024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa045eae84748a59afc512c822816b6, State: Initialized, Role: FOLLOWER
I20240502 20:45:45.811969 23024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa045eae84748a59afc512c822816b6" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 46265 } }
I20240502 20:45:45.812117 23024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:45.812183 23024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:45.812309 23024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:45.813375 23024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa045eae84748a59afc512c822816b6" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 46265 } }
I20240502 20:45:45.813774 23024 leader_election.cc:302] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aa045eae84748a59afc512c822816b6; no voters: 
I20240502 20:45:45.814095 23024 leader_election.cc:288] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 20:45:45.814819 23029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:45.815035 23029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa045eae84748a59afc512c822816b6, State: Running, Role: LEADER
I20240502 20:45:45.815339 23024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:45.815363 23029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa045eae84748a59afc512c822816b6" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 46265 } }
I20240502 20:45:45.817093 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aa045eae84748a59afc512c822816b6. Latest consensus state: current_term: 1 leader_uuid: "2aa045eae84748a59afc512c822816b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa045eae84748a59afc512c822816b6" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 46265 } } }
I20240502 20:45:45.817248 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:45.817157 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aa045eae84748a59afc512c822816b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa045eae84748a59afc512c822816b6" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 46265 } } }
I20240502 20:45:45.817301 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:45.817646 23038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:45.821399 23038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:45.829214 23038 catalog_manager.cc:1347] Generated new cluster ID: 32337e37e2aa4ead8661284d0aa808e4
I20240502 20:45:45.829350 23038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:45.840248 23038 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:45.841908 23038 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:45.849817 23038 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2aa045eae84748a59afc512c822816b6: Generated new TSK 0
I20240502 20:45:45.850826 23038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:45.858033 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.129:0
--local_ip_for_outbound_sockets=127.19.66.129
--webserver_interface=127.19.66.129
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:45.999339 23048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:45.999656 23048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:45.999707 23048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:46.003800 23048 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:46.003921 23048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:46.004004 23048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.129
I20240502 20:45:46.008734 23048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.66.129
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.23048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.129
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:46.009894 23048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:46.011197 23048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:46.022318 23053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:46.022886 23056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:46.023706 23054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:46.024397 23048 server_base.cc:1034] running on GCE node
I20240502 20:45:46.024936 23048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:46.025671 23048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:46.026872 23048 hybrid_clock.cc:648] HybridClock initialized: now 1714682746026815 us; error 74 us; skew 500 ppm
I20240502 20:45:46.029304 23048 webserver.cc:460] Webserver started at http://127.19.66.129:35225/ using document root <none> and password file <none>
I20240502 20:45:46.030076 23048 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:46.030164 23048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:46.030370 23048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:46.032250 23048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "140dd1ea33be47e99530c773c5439d56"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.032750 23048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "140dd1ea33be47e99530c773c5439d56"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.036608 23048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20240502 20:45:46.039279 23062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.040472 23048 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20240502 20:45:46.040607 23048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "140dd1ea33be47e99530c773c5439d56"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.040727 23048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:46.058917 23048 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:46.059782 23048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:46.059967 23048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:46.060617 23048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:46.061686 23048 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:46.061739 23048 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.061816 23048 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:46.061851 23048 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.072476 23048 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.129:45439
I20240502 20:45:46.072512 23179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.129:45439 every 8 connection(s)
I20240502 20:45:46.073580 23048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:45:46.077056 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 23048
I20240502 20:45:46.077239 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:45:46.079589 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.130:0
--local_ip_for_outbound_sockets=127.19.66.130
--webserver_interface=127.19.66.130
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:46.085952 23180 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:46.086269 23180 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:46.087092 23180 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:46.088898 22987 ts_manager.cc:194] Registered new tserver with Master: 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:46.090286 22987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.129:53521
W20240502 20:45:46.210481 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:46.210795 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:46.210884 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:46.214453 23183 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:46.214560 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:46.214689 23183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.130
I20240502 20:45:46.219662 23183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.66.130
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.23183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.130
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:46.220777 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:46.222097 23183 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:46.232419 23191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:46.233084 23189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:46.233741 23188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:46.234587 23183 server_base.cc:1034] running on GCE node
I20240502 20:45:46.235127 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:46.235873 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:46.237104 23183 hybrid_clock.cc:648] HybridClock initialized: now 1714682746237046 us; error 46 us; skew 500 ppm
I20240502 20:45:46.240660 23183 webserver.cc:460] Webserver started at http://127.19.66.130:43075/ using document root <none> and password file <none>
I20240502 20:45:46.241786 23183 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:46.241863 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:46.242079 23183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:46.244139 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3db3750c037e4393927784c55f66d349"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.244765 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3db3750c037e4393927784c55f66d349"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.249217 23183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.008s	sys 0.000s
I20240502 20:45:46.252074 23197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.253516 23183 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:46.253741 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3db3750c037e4393927784c55f66d349"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.253885 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:46.283921 23183 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:46.285156 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:46.285409 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:46.286115 23183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:46.287253 23183 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:46.287317 23183 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.287389 23183 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:46.287420 23183 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.298228 23183 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.130:35415
I20240502 20:45:46.298473 23314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.130:35415 every 8 connection(s)
I20240502 20:45:46.299562 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:45:46.309118 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 23183
I20240502 20:45:46.309286 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:45:46.312120 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.131:0
--local_ip_for_outbound_sockets=127.19.66.131
--webserver_interface=127.19.66.131
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:46.312536 23315 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:46.312815 23315 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:46.313488 23315 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:46.315150 22987 ts_manager.cc:194] Registered new tserver with Master: 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:46.315884 22987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.130:44673
W20240502 20:45:46.439270 23318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:46.439553 23318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:46.439635 23318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:46.443158 23318 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:46.443264 23318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:46.443382 23318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.131
I20240502 20:45:46.447654 23318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.66.131
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.23318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.131
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:46.448809 23318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:46.450188 23318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:46.460026 23324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:46.460666 23326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:46.461365 23323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:46.462154 23318 server_base.cc:1034] running on GCE node
I20240502 20:45:46.462677 23318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:46.463354 23318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:46.464543 23318 hybrid_clock.cc:648] HybridClock initialized: now 1714682746464533 us; error 30 us; skew 500 ppm
I20240502 20:45:46.467060 23318 webserver.cc:460] Webserver started at http://127.19.66.131:44973/ using document root <none> and password file <none>
I20240502 20:45:46.467722 23318 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:46.467803 23318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:46.468034 23318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:46.470053 23318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ee6fe9bcf23a450cb5432a8205660598"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.470602 23318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ee6fe9bcf23a450cb5432a8205660598"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.474943 23318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 20:45:46.478597 23332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.480432 23318 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:45:46.480600 23318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ee6fe9bcf23a450cb5432a8205660598"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.480744 23318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:46.492620 23318 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:46.493446 23318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:46.493630 23318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:46.494232 23318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:46.495222 23318 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:46.495285 23318 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.495352 23318 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:46.495378 23318 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.505632 23318 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.131:39261
I20240502 20:45:46.505728 23450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.131:39261 every 8 connection(s)
I20240502 20:45:46.506703 23318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:45:46.510705 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 23318
I20240502 20:45:46.510835 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:45:46.513671 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.132:0
--local_ip_for_outbound_sockets=127.19.66.132
--webserver_interface=127.19.66.132
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:46.518534 23451 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:46.518882 23451 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:46.519805 23451 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:46.521389 22987 ts_manager.cc:194] Registered new tserver with Master: ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:46.522037 22987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.131:37261
W20240502 20:45:46.642292 23454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:46.642575 23454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:46.642625 23454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:46.646176 23454 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:46.646268 23454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:46.646349 23454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.132
I20240502 20:45:46.650432 23454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.66.132
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.23454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.132
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:46.651585 23454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:46.652874 23454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:46.663671 23460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:46.664075 23462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:46.664731 23459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:46.665540 23454 server_base.cc:1034] running on GCE node
I20240502 20:45:46.666080 23454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:46.666808 23454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:46.668071 23454 hybrid_clock.cc:648] HybridClock initialized: now 1714682746668023 us; error 49 us; skew 500 ppm
I20240502 20:45:46.671097 23454 webserver.cc:460] Webserver started at http://127.19.66.132:37929/ using document root <none> and password file <none>
I20240502 20:45:46.672021 23454 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:46.672102 23454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:46.672586 23454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:46.674837 23454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "94ac80c7dd7f4767bb357b89c547d8b2"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.675454 23454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "94ac80c7dd7f4767bb357b89c547d8b2"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.680132 23454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.003s
I20240502 20:45:46.683524 23468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.684978 23454 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:46.685176 23454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "94ac80c7dd7f4767bb357b89c547d8b2"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.685343 23454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:46.700501 23454 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:46.701409 23454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:46.701615 23454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:46.702278 23454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:46.703632 23454 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:46.703699 23454 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.703775 23454 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:46.703809 23454 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.713760 23454 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.132:45001
I20240502 20:45:46.713858 23583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.132:45001 every 8 connection(s)
I20240502 20:45:46.714872 23454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:46.722600 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 23454
I20240502 20:45:46.722759 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:45:46.725850 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.133:0
--local_ip_for_outbound_sockets=127.19.66.133
--webserver_interface=127.19.66.133
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:46.727859 23584 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:46.728206 23584 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:46.728857 23584 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:46.730348 22987 ts_manager.cc:194] Registered new tserver with Master: 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132:45001)
I20240502 20:45:46.731074 22987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.132:45349
W20240502 20:45:46.855623 23587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:46.855897 23587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:46.855949 23587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:46.859845 23587 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:46.859942 23587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:46.860028 23587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.133
I20240502 20:45:46.864264 23587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.66.133
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.23587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.133
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:46.865543 23587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:46.866879 23587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:46.877566 23592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:46.878101 23595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:46.878659 23593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:46.879433 23587 server_base.cc:1034] running on GCE node
I20240502 20:45:46.879982 23587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:46.880738 23587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:46.881922 23587 hybrid_clock.cc:648] HybridClock initialized: now 1714682746881907 us; error 33 us; skew 500 ppm
I20240502 20:45:46.884552 23587 webserver.cc:460] Webserver started at http://127.19.66.133:45695/ using document root <none> and password file <none>
I20240502 20:45:46.885309 23587 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:46.885380 23587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:46.885594 23587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:46.887545 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b30503059718458bb016185550f5e0fb"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.888113 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b30503059718458bb016185550f5e0fb"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.893602 23587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20240502 20:45:46.897186 23601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.898548 23587 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 20:45:46.898707 23587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b30503059718458bb016185550f5e0fb"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:46.898847 23587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:46.910230 23587 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:46.911002 23587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:46.911186 23587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:46.911820 23587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:46.912833 23587 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:46.912887 23587 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.912951 23587 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:46.912983 23587 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:46.924290 23587 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.133:45729
I20240502 20:45:46.924322 23718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.133:45729 every 8 connection(s)
I20240502 20:45:46.925451 23587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:46.935727 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 23587
I20240502 20:45:46.935865 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:45:46.936527 23719 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:46.936969 23719 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:46.938037 23719 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:46.938810 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.134:0
--local_ip_for_outbound_sockets=127.19.66.134
--webserver_interface=127.19.66.134
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:46.939368 22987 ts_manager.cc:194] Registered new tserver with Master: b30503059718458bb016185550f5e0fb (127.19.66.133:45729)
I20240502 20:45:46.940006 22987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.133:34907
W20240502 20:45:47.067740 23722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:47.068009 23722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:47.068060 23722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:47.071419 23722 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:47.071503 23722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:47.071580 23722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.134
I20240502 20:45:47.076279 23722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.19.66.134
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.23722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.134
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:47.077399 23722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:47.078924 23722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:47.089449 23730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:47.089869 23727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:47.090368 23728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:47.091245 23722 server_base.cc:1034] running on GCE node
I20240502 20:45:47.091814 23722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:47.092521 23722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:47.093006 23180 heartbeater.cc:503] Master 127.19.66.190:46265 was elected leader, sending a full tablet report...
I20240502 20:45:47.093689 23722 hybrid_clock.cc:648] HybridClock initialized: now 1714682747093678 us; error 31 us; skew 500 ppm
I20240502 20:45:47.096182 23722 webserver.cc:460] Webserver started at http://127.19.66.134:43609/ using document root <none> and password file <none>
I20240502 20:45:47.096858 23722 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:47.096946 23722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:47.097214 23722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:47.099179 23722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c9af0b94bfe2492bb34c7255879a7328"
format_stamp: "Formatted at 2024-05-02 20:45:47 on dist-test-slave-gqqg"
I20240502 20:45:47.099718 23722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c9af0b94bfe2492bb34c7255879a7328"
format_stamp: "Formatted at 2024-05-02 20:45:47 on dist-test-slave-gqqg"
I20240502 20:45:47.103682 23722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20240502 20:45:47.106446 23736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:47.107851 23722 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:45:47.107986 23722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c9af0b94bfe2492bb34c7255879a7328"
format_stamp: "Formatted at 2024-05-02 20:45:47 on dist-test-slave-gqqg"
I20240502 20:45:47.108114 23722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:47.135623 23722 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:47.136430 23722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:47.136618 23722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:47.137276 23722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:47.138363 23722 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:47.138413 23722 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:47.138466 23722 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:47.138497 23722 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:47.148140 23722 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.134:43861
I20240502 20:45:47.148224 23849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.134:43861 every 8 connection(s)
I20240502 20:45:47.149237 23722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:45:47.158372 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 23722
I20240502 20:45:47.158542 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 20:45:47.159183 23850 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:47.159478 23850 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:47.160151 23850 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:47.160712 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.135:0
--local_ip_for_outbound_sockets=127.19.66.135
--webserver_interface=127.19.66.135
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:47.161252 22987 ts_manager.cc:194] Registered new tserver with Master: c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861)
I20240502 20:45:47.161824 22987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.134:35839
W20240502 20:45:47.291604 23853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:47.291890 23853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:47.291945 23853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:47.296170 23853 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:47.296408 23853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:47.296589 23853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.135
I20240502 20:45:47.302191 23853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.19.66.135
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.23853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.135
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:47.303256 23853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:47.304452 23853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:47.314358 23859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:47.314700 23858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:47.315474 23861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:47.316560 23853 server_base.cc:1034] running on GCE node
I20240502 20:45:47.317131 23853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:47.317904 23853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:47.319082 23853 hybrid_clock.cc:648] HybridClock initialized: now 1714682747319059 us; error 47 us; skew 500 ppm
I20240502 20:45:47.319100 23315 heartbeater.cc:503] Master 127.19.66.190:46265 was elected leader, sending a full tablet report...
I20240502 20:45:47.321853 23853 webserver.cc:460] Webserver started at http://127.19.66.135:45889/ using document root <none> and password file <none>
I20240502 20:45:47.322489 23853 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:47.322559 23853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:47.322741 23853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:47.324770 23853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "09f2f86617004c96b1e186759506cc68"
format_stamp: "Formatted at 2024-05-02 20:45:47 on dist-test-slave-gqqg"
I20240502 20:45:47.325438 23853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "09f2f86617004c96b1e186759506cc68"
format_stamp: "Formatted at 2024-05-02 20:45:47 on dist-test-slave-gqqg"
I20240502 20:45:47.329524 23853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240502 20:45:47.332178 23867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:47.333976 23853 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:47.334122 23853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "09f2f86617004c96b1e186759506cc68"
format_stamp: "Formatted at 2024-05-02 20:45:47 on dist-test-slave-gqqg"
I20240502 20:45:47.334230 23853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:47.355717 23853 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:47.356457 23853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:47.356621 23853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:47.357220 23853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:47.358237 23853 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:47.358289 23853 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:47.358332 23853 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:47.358353 23853 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:47.368074 23853 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.135:35917
I20240502 20:45:47.368167 23983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.135:35917 every 8 connection(s)
I20240502 20:45:47.369421 23853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:45:47.378450 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 23853
I20240502 20:45:47.378610 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 20:45:47.379693 23984 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:47.379971 23984 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:47.380637 23984 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:47.381814 22987 ts_manager.cc:194] Registered new tserver with Master: 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917)
I20240502 20:45:47.382438 22987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.135:38885
I20240502 20:45:47.394016 19722 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 20:45:47.405110 22987 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41988:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 20:45:47.426017 23916 tablet_service.cc:1467] Processing CreateTablet for tablet 4d54cbc40bcc4fa481a04d100ed3c927 (DEFAULT_TABLE table=TestTable [id=00f7d8412486460a969b80ac0903c676]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:47.426017 23516 tablet_service.cc:1467] Processing CreateTablet for tablet 4d54cbc40bcc4fa481a04d100ed3c927 (DEFAULT_TABLE table=TestTable [id=00f7d8412486460a969b80ac0903c676]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:47.427377 23516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d54cbc40bcc4fa481a04d100ed3c927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:47.427377 23916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d54cbc40bcc4fa481a04d100ed3c927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:47.433699 24000 tablet_bootstrap.cc:492] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap starting.
I20240502 20:45:47.433714 24001 tablet_bootstrap.cc:492] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Bootstrap starting.
I20240502 20:45:47.436228 24000 tablet_bootstrap.cc:654] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:47.437005 24000 log.cc:824] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:47.437376 24001 tablet_bootstrap.cc:654] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:47.438118 24001 log.cc:824] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:47.439083 24000 tablet_bootstrap.cc:492] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: No bootstrap required, opened a new log
I20240502 20:45:47.439281 24000 ts_tablet_manager.cc:1367] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.004s
I20240502 20:45:47.439754 24001 tablet_bootstrap.cc:492] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: No bootstrap required, opened a new log
I20240502 20:45:47.439918 24001 ts_tablet_manager.cc:1367] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 20:45:47.442219 24000 raft_consensus.cc:348] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:47.442516 24000 raft_consensus.cc:374] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:47.442600 24000 raft_consensus.cc:729] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ac80c7dd7f4767bb357b89c547d8b2, State: Initialized, Role: FOLLOWER
I20240502 20:45:47.443040 24000 consensus_queue.cc:260] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:47.443337 24001 raft_consensus.cc:348] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:47.443624 24001 raft_consensus.cc:374] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:47.443686 24001 raft_consensus.cc:729] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Initialized, Role: FOLLOWER
I20240502 20:45:47.443992 24000 ts_tablet_manager.cc:1398] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.001s
I20240502 20:45:47.444051 23584 heartbeater.cc:503] Master 127.19.66.190:46265 was elected leader, sending a full tablet report...
I20240502 20:45:47.445294 24001 consensus_queue.cc:260] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:47.446084 23984 heartbeater.cc:503] Master 127.19.66.190:46265 was elected leader, sending a full tablet report...
I20240502 20:45:47.446238 24001 ts_tablet_manager.cc:1398] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
W20240502 20:45:47.466444 23585 tablet.cc:2369] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:47.525168 23451 heartbeater.cc:503] Master 127.19.66.190:46265 was elected leader, sending a full tablet report...
W20240502 20:45:47.620564 23985 tablet.cc:2369] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:47.684023 24005 raft_consensus.cc:482] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:47.684259 24005 raft_consensus.cc:504] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:47.685654 24005 leader_election.cc:288] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132:45001)
I20240502 20:45:47.689513 23537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d54cbc40bcc4fa481a04d100ed3c927" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" is_pre_election: true
I20240502 20:45:47.689846 23537 raft_consensus.cc:2454] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 0.
I20240502 20:45:47.690392 23871 leader_election.cc:302] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09f2f86617004c96b1e186759506cc68, 94ac80c7dd7f4767bb357b89c547d8b2; no voters: 
I20240502 20:45:47.690833 24005 raft_consensus.cc:2781] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:47.691017 24005 raft_consensus.cc:482] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:47.691076 24005 raft_consensus.cc:3037] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:47.692297 24005 raft_consensus.cc:504] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:47.692719 24005 leader_election.cc:288] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 1 election: Requested vote from peers 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132:45001)
I20240502 20:45:47.693372 23537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d54cbc40bcc4fa481a04d100ed3c927" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ac80c7dd7f4767bb357b89c547d8b2"
I20240502 20:45:47.693511 23537 raft_consensus.cc:3037] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:47.694703 23537 raft_consensus.cc:2454] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 1.
I20240502 20:45:47.695169 23871 leader_election.cc:302] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09f2f86617004c96b1e186759506cc68, 94ac80c7dd7f4767bb357b89c547d8b2; no voters: 
I20240502 20:45:47.695431 24005 raft_consensus.cc:2781] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:47.695749 24005 raft_consensus.cc:686] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 LEADER]: Becoming Leader. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Running, Role: LEADER
I20240502 20:45:47.696139 24005 consensus_queue.cc:237] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:47.700016 22986 catalog_manager.cc:5484] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f2f86617004c96b1e186759506cc68 (127.19.66.135). New cstate: current_term: 1 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:47.719170 19722 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 20:45:47.720898 19722 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 140dd1ea33be47e99530c773c5439d56 to finish bootstrapping
I20240502 20:45:47.729499 19722 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3db3750c037e4393927784c55f66d349 to finish bootstrapping
I20240502 20:45:47.736442 19722 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ee6fe9bcf23a450cb5432a8205660598 to finish bootstrapping
I20240502 20:45:47.744545 19722 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 94ac80c7dd7f4767bb357b89c547d8b2 to finish bootstrapping
I20240502 20:45:47.751401 19722 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b30503059718458bb016185550f5e0fb to finish bootstrapping
I20240502 20:45:47.759665 19722 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c9af0b94bfe2492bb34c7255879a7328 to finish bootstrapping
I20240502 20:45:47.769683 19722 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 09f2f86617004c96b1e186759506cc68 to finish bootstrapping
I20240502 20:45:47.778545 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 23454
I20240502 20:45:47.786073 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 23587
I20240502 20:45:47.792934 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 23722
I20240502 20:45:47.801721 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 23853
I20240502 20:45:51.561520 22986 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41988:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:51.580861 23381 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab508cf18c46d8ad1775086354f379 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b45b0775e1d8423585dd5d5a47ec5dbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:51.580861 23110 tablet_service.cc:1467] Processing CreateTablet for tablet 96ee320aaf2c40a9885f934c5c653c78 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b45b0775e1d8423585dd5d5a47ec5dbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:51.580911 23246 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab508cf18c46d8ad1775086354f379 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b45b0775e1d8423585dd5d5a47ec5dbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:51.581846 23247 tablet_service.cc:1467] Processing CreateTablet for tablet 96ee320aaf2c40a9885f934c5c653c78 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b45b0775e1d8423585dd5d5a47ec5dbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:51.581372 23382 tablet_service.cc:1467] Processing CreateTablet for tablet ddfa7e208cb1404ba25ebc77a75f7c1a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b45b0775e1d8423585dd5d5a47ec5dbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:51.582424 23110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ee320aaf2c40a9885f934c5c653c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:51.582422 23246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab508cf18c46d8ad1775086354f379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:51.582485 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddfa7e208cb1404ba25ebc77a75f7c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:51.580861 23113 tablet_service.cc:1467] Processing CreateTablet for tablet ddfa7e208cb1404ba25ebc77a75f7c1a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b45b0775e1d8423585dd5d5a47ec5dbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:51.582861 23247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ee320aaf2c40a9885f934c5c653c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:51.584762 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab508cf18c46d8ad1775086354f379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:51.584478 23113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddfa7e208cb1404ba25ebc77a75f7c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:51.593195 24025 tablet_bootstrap.cc:492] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349: Bootstrap starting.
I20240502 20:45:51.593456 24026 tablet_bootstrap.cc:492] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598: Bootstrap starting.
I20240502 20:45:51.594621 24027 tablet_bootstrap.cc:492] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: Bootstrap starting.
I20240502 20:45:51.596473 24026 tablet_bootstrap.cc:654] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:51.596473 24025 tablet_bootstrap.cc:654] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:51.596971 24027 tablet_bootstrap.cc:654] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:51.597640 24026 log.cc:824] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:51.597640 24025 log.cc:824] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:51.597829 24027 log.cc:824] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:51.599984 24027 tablet_bootstrap.cc:492] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: No bootstrap required, opened a new log
I20240502 20:45:51.599984 24026 tablet_bootstrap.cc:492] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598: No bootstrap required, opened a new log
I20240502 20:45:51.599984 24025 tablet_bootstrap.cc:492] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349: No bootstrap required, opened a new log
I20240502 20:45:51.600231 24027 ts_tablet_manager.cc:1367] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 20:45:51.600232 24026 ts_tablet_manager.cc:1367] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 20:45:51.600232 24025 ts_tablet_manager.cc:1367] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 20:45:51.604720 24027 raft_consensus.cc:348] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:51.604815 24025 raft_consensus.cc:348] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:51.604815 24026 raft_consensus.cc:348] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:51.605039 24025 raft_consensus.cc:374] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:51.605039 24026 raft_consensus.cc:374] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:51.605028 24027 raft_consensus.cc:374] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:51.605144 24026 raft_consensus.cc:729] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Initialized, Role: FOLLOWER
I20240502 20:45:51.605144 24027 raft_consensus.cc:729] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Initialized, Role: FOLLOWER
I20240502 20:45:51.605190 24025 raft_consensus.cc:729] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Initialized, Role: FOLLOWER
I20240502 20:45:51.605778 24027 consensus_queue.cc:260] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:51.605778 24026 consensus_queue.cc:260] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:51.605778 24025 consensus_queue.cc:260] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:51.607672 24027 ts_tablet_manager.cc:1398] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20240502 20:45:51.607038 24026 ts_tablet_manager.cc:1398] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.004s
I20240502 20:45:51.608237 24025 ts_tablet_manager.cc:1398] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20240502 20:45:51.608693 24027 tablet_bootstrap.cc:492] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56: Bootstrap starting.
I20240502 20:45:51.609130 24025 tablet_bootstrap.cc:492] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: Bootstrap starting.
I20240502 20:45:51.609681 24026 tablet_bootstrap.cc:492] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: Bootstrap starting.
I20240502 20:45:51.610551 24025 tablet_bootstrap.cc:654] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:51.612264 24027 tablet_bootstrap.cc:654] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:51.612592 24026 tablet_bootstrap.cc:654] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:51.613636 24025 tablet_bootstrap.cc:492] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: No bootstrap required, opened a new log
I20240502 20:45:51.613795 24025 ts_tablet_manager.cc:1367] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 20:45:51.613819 24027 tablet_bootstrap.cc:492] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56: No bootstrap required, opened a new log
I20240502 20:45:51.613902 24027 ts_tablet_manager.cc:1367] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.001s
I20240502 20:45:51.614622 24025 raft_consensus.cc:348] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:51.614622 24027 raft_consensus.cc:348] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:51.614773 24027 raft_consensus.cc:374] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:51.614773 24025 raft_consensus.cc:374] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:51.614809 24027 raft_consensus.cc:729] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Initialized, Role: FOLLOWER
I20240502 20:45:51.614809 24025 raft_consensus.cc:729] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Initialized, Role: FOLLOWER
I20240502 20:45:51.614961 24027 consensus_queue.cc:260] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:51.614961 24025 consensus_queue.cc:260] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:51.615293 24027 ts_tablet_manager.cc:1398] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
I20240502 20:45:51.615293 24025 ts_tablet_manager.cc:1398] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:51.616299 24026 tablet_bootstrap.cc:492] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: No bootstrap required, opened a new log
I20240502 20:45:51.616505 24026 ts_tablet_manager.cc:1367] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.003s
I20240502 20:45:51.617372 24026 raft_consensus.cc:348] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:51.617538 24026 raft_consensus.cc:374] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:51.617575 24026 raft_consensus.cc:729] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Initialized, Role: FOLLOWER
I20240502 20:45:51.617753 24026 consensus_queue.cc:260] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:51.618052 24026 ts_tablet_manager.cc:1398] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:51.650270 24033 raft_consensus.cc:482] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:51.650576 24033 raft_consensus.cc:504] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:51.652076 24033 leader_election.cc:288] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:51.657310 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ee320aaf2c40a9885f934c5c653c78" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56" is_pre_election: true
I20240502 20:45:51.657783 23133 raft_consensus.cc:2454] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 0.
I20240502 20:45:51.658324 23200 leader_election.cc:302] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:51.658640 24033 raft_consensus.cc:2781] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:51.658735 24033 raft_consensus.cc:482] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:51.658798 24033 raft_consensus.cc:3037] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:51.660307 24033 raft_consensus.cc:504] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:51.660754 24033 leader_election.cc:288] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Requested vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:51.661286 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ee320aaf2c40a9885f934c5c653c78" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56"
I20240502 20:45:51.661502 23133 raft_consensus.cc:3037] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:51.662715 23133 raft_consensus.cc:2454] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 1.
I20240502 20:45:51.663107 23200 leader_election.cc:302] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:51.663379 24033 raft_consensus.cc:2781] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:51.663668 24033 raft_consensus.cc:686] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Becoming Leader. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:45:51.664126 24033 consensus_queue.cc:237] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:51.668293 22986 catalog_manager.cc:5484] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db3750c037e4393927784c55f66d349 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:51.680761 24033 raft_consensus.cc:482] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:51.680969 24033 raft_consensus.cc:504] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:51.681700 24033 leader_election.cc:288] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:51.685986 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598" is_pre_election: true
I20240502 20:45:51.686466 23404 raft_consensus.cc:2454] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 0.
I20240502 20:45:51.686978 23199 leader_election.cc:302] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:51.687222 24033 raft_consensus.cc:2781] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:51.687314 24033 raft_consensus.cc:482] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:51.687348 24033 raft_consensus.cc:3037] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:51.688426 24033 raft_consensus.cc:504] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:51.688825 24033 leader_election.cc:288] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Requested vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:51.689440 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
I20240502 20:45:51.689617 23404 raft_consensus.cc:3037] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:51.690809 23404 raft_consensus.cc:2454] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 1.
I20240502 20:45:51.691185 23199 leader_election.cc:302] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:51.691386 24033 raft_consensus.cc:2781] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:51.691486 24033 raft_consensus.cc:686] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Becoming Leader. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:45:51.691674 24033 consensus_queue.cc:237] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:51.693987 22986 catalog_manager.cc:5484] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db3750c037e4393927784c55f66d349 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:51.763568 23452 tablet.cc:2369] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:51.808331 23316 tablet.cc:2369] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:51.832748 23181 tablet.cc:2369] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:51.850904 24032 raft_consensus.cc:482] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:51.851130 24032 raft_consensus.cc:504] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:51.852532 24032 leader_election.cc:288] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:51.856904 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfa7e208cb1404ba25ebc77a75f7c1a" candidate_uuid: "ee6fe9bcf23a450cb5432a8205660598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56" is_pre_election: true
I20240502 20:45:51.857183 23133 raft_consensus.cc:2454] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee6fe9bcf23a450cb5432a8205660598 in term 0.
I20240502 20:45:51.857728 23334 leader_election.cc:302] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:51.858094 24032 raft_consensus.cc:2781] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:51.858172 24032 raft_consensus.cc:482] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:51.858206 24032 raft_consensus.cc:3037] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:51.859398 24032 raft_consensus.cc:504] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:51.859797 24032 leader_election.cc:288] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 election: Requested vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:51.860458 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfa7e208cb1404ba25ebc77a75f7c1a" candidate_uuid: "ee6fe9bcf23a450cb5432a8205660598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56"
I20240502 20:45:51.860622 23133 raft_consensus.cc:3037] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:51.861721 23133 raft_consensus.cc:2454] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee6fe9bcf23a450cb5432a8205660598 in term 1.
I20240502 20:45:51.862140 23334 leader_election.cc:302] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:51.862366 24032 raft_consensus.cc:2781] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:51.862653 24032 raft_consensus.cc:686] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Becoming Leader. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Running, Role: LEADER
I20240502 20:45:51.863135 24032 consensus_queue.cc:237] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:51.867597 22986 catalog_manager.cc:5484] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:52.015465 24044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:52.015832 24044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:52.019515 24044 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 20:45:52.087807 24033 consensus_queue.cc:1035] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:52.136801 23133 raft_consensus.cc:1261] T ddfa7e208cb1404ba25ebc77a75f7c1a P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:52.138329 23404 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:52.138988 24033 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:52.145743 24032 consensus_queue.cc:1035] T ddfa7e208cb1404ba25ebc77a75f7c1a P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:52.146378 24068 mvcc.cc:205] Tried to move back new op lower bound from 7023340552747773952 to 7023340550930526208. Current Snapshot: MvccSnapshot[applied={T|T < 7023340552747773952}]
I20240502 20:45:52.171149 24084 mvcc.cc:205] Tried to move back new op lower bound from 7023340552763437056 to 7023340551634669568. Current Snapshot: MvccSnapshot[applied={T|T < 7023340552744329216}]

INSERT report
    rows total: 2000
    time total: 113.777 ms
  time per row: 0.0568883 ms
I20240502 20:45:52.195587 22985 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41988:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:52.204998 23110 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:52.205367 23381 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d185d01e44dd49580eeb4ab979346 (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:52.205406 23113 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d185d01e44dd49580eeb4ab979346 (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:52.205657 23110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6c6fc06ec341e28a4b9f231d78ac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:52.205710 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d185d01e44dd49580eeb4ab979346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:52.205956 23113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d185d01e44dd49580eeb4ab979346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:52.206121 23382 tablet_service.cc:1467] Processing CreateTablet for tablet b0f06fbb9f58417793a6d176b1f1078f (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:52.206365 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f06fbb9f58417793a6d176b1f1078f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:52.208648 24093 tablet_bootstrap.cc:492] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598: Bootstrap starting.
I20240502 20:45:52.208874 23246 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:52.209234 23246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6c6fc06ec341e28a4b9f231d78ac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:52.209528 24092 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56: Bootstrap starting.
I20240502 20:45:52.209810 24093 tablet_bootstrap.cc:654] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:52.210707 24092 tablet_bootstrap.cc:654] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:52.210748 23247 tablet_service.cc:1467] Processing CreateTablet for tablet b0f06fbb9f58417793a6d176b1f1078f (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:52.211016 23247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f06fbb9f58417793a6d176b1f1078f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:52.211884 24093 tablet_bootstrap.cc:492] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598: No bootstrap required, opened a new log
I20240502 20:45:52.211985 24093 ts_tablet_manager.cc:1367] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:52.212112 24096 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: Bootstrap starting.
I20240502 20:45:52.212512 24093 raft_consensus.cc:348] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:52.212622 24093 raft_consensus.cc:374] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:52.212663 24093 raft_consensus.cc:729] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Initialized, Role: FOLLOWER
I20240502 20:45:52.212837 24093 consensus_queue.cc:260] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:52.213249 24096 tablet_bootstrap.cc:654] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:52.213378 24093 ts_tablet_manager.cc:1398] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:52.213539 24093 tablet_bootstrap.cc:492] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598: Bootstrap starting.
I20240502 20:45:52.214262 24092 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56: No bootstrap required, opened a new log
I20240502 20:45:52.214365 24092 ts_tablet_manager.cc:1367] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:52.214725 24093 tablet_bootstrap.cc:654] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:52.214944 24092 raft_consensus.cc:348] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:52.215060 24092 raft_consensus.cc:374] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:52.215070 24096 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: No bootstrap required, opened a new log
I20240502 20:45:52.215101 24092 raft_consensus.cc:729] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Initialized, Role: FOLLOWER
I20240502 20:45:52.215155 24096 ts_tablet_manager.cc:1367] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:52.215265 24092 consensus_queue.cc:260] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:52.215667 24092 ts_tablet_manager.cc:1398] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:52.215709 24096 raft_consensus.cc:348] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:52.215814 24096 raft_consensus.cc:374] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:52.215847 24096 raft_consensus.cc:729] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Initialized, Role: FOLLOWER
I20240502 20:45:52.215899 24092 tablet_bootstrap.cc:492] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56: Bootstrap starting.
I20240502 20:45:52.216012 24096 consensus_queue.cc:260] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:52.216295 24096 ts_tablet_manager.cc:1398] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:52.216512 24096 tablet_bootstrap.cc:492] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349: Bootstrap starting.
I20240502 20:45:52.217736 24096 tablet_bootstrap.cc:654] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:52.218538 24093 tablet_bootstrap.cc:492] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598: No bootstrap required, opened a new log
I20240502 20:45:52.218643 24093 ts_tablet_manager.cc:1367] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:45:52.219002 24092 tablet_bootstrap.cc:654] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:52.219154 24093 raft_consensus.cc:348] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:52.219283 24093 raft_consensus.cc:374] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:52.219319 24093 raft_consensus.cc:729] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Initialized, Role: FOLLOWER
I20240502 20:45:52.219450 24093 consensus_queue.cc:260] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:52.219662 24096 tablet_bootstrap.cc:492] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349: No bootstrap required, opened a new log
I20240502 20:45:52.219774 24093 ts_tablet_manager.cc:1398] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:52.220106 24096 ts_tablet_manager.cc:1367] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 20:45:52.220937 24092 tablet_bootstrap.cc:492] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56: No bootstrap required, opened a new log
I20240502 20:45:52.220891 24096 raft_consensus.cc:348] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:52.221029 24096 raft_consensus.cc:374] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:52.221096 24096 raft_consensus.cc:729] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Initialized, Role: FOLLOWER
I20240502 20:45:52.221122 24092 ts_tablet_manager.cc:1367] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 20:45:52.221617 24096 consensus_queue.cc:260] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:52.221755 24092 raft_consensus.cc:348] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:52.221869 24092 raft_consensus.cc:374] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:52.221902 24092 raft_consensus.cc:729] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Initialized, Role: FOLLOWER
I20240502 20:45:52.222076 24096 ts_tablet_manager.cc:1398] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:52.222072 24092 consensus_queue.cc:260] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:52.222397 24092 ts_tablet_manager.cc:1398] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:52.332567 24033 raft_consensus.cc:482] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:52.332773 24033 raft_consensus.cc:504] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:52.333315 24033 leader_election.cc:288] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:52.334231 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f06fbb9f58417793a6d176b1f1078f" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598" is_pre_election: true
I20240502 20:45:52.334506 23404 raft_consensus.cc:2454] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 0.
I20240502 20:45:52.335000 23199 leader_election.cc:302] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:52.335402 24033 raft_consensus.cc:2781] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:52.335465 24033 raft_consensus.cc:482] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:52.335506 24033 raft_consensus.cc:3037] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:52.336447 24033 raft_consensus.cc:504] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:52.336807 24033 leader_election.cc:288] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Requested vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:52.337729 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f06fbb9f58417793a6d176b1f1078f" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
I20240502 20:45:52.337881 23404 raft_consensus.cc:3037] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:52.338894 23404 raft_consensus.cc:2454] T b0f06fbb9f58417793a6d176b1f1078f P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 1.
I20240502 20:45:52.339552 23199 leader_election.cc:302] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:52.339951 24033 raft_consensus.cc:2781] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:52.340068 24033 raft_consensus.cc:686] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Becoming Leader. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:45:52.340271 24033 consensus_queue.cc:237] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:52.342772 22985 catalog_manager.cc:5484] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db3750c037e4393927784c55f66d349 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:52.385278 24031 raft_consensus.cc:482] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:52.385470 24031 raft_consensus.cc:504] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:52.386713 24031 leader_election.cc:288] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:52.390928 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d185d01e44dd49580eeb4ab979346" candidate_uuid: "140dd1ea33be47e99530c773c5439d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598" is_pre_election: true
I20240502 20:45:52.391108 23404 raft_consensus.cc:2454] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140dd1ea33be47e99530c773c5439d56 in term 0.
I20240502 20:45:52.391470 23063 leader_election.cc:302] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:52.391757 24031 raft_consensus.cc:2781] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:52.391831 24031 raft_consensus.cc:482] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:52.391881 24031 raft_consensus.cc:3037] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:52.392882 24031 raft_consensus.cc:504] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:52.393646 24031 leader_election.cc:288] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 election: Requested vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:52.394225 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d185d01e44dd49580eeb4ab979346" candidate_uuid: "140dd1ea33be47e99530c773c5439d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
I20240502 20:45:52.394389 23404 raft_consensus.cc:3037] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:52.395577 23404 raft_consensus.cc:2454] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140dd1ea33be47e99530c773c5439d56 in term 1.
I20240502 20:45:52.396137 23063 leader_election.cc:302] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:52.396447 24031 raft_consensus.cc:2781] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:52.396770 24031 raft_consensus.cc:686] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Becoming Leader. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Running, Role: LEADER
I20240502 20:45:52.397230 24031 consensus_queue.cc:237] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:52.400481 22985 catalog_manager.cc:5484] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 140dd1ea33be47e99530c773c5439d56 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:52.676975 24033 raft_consensus.cc:482] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:52.677213 24033 raft_consensus.cc:504] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:52.677827 24033 leader_election.cc:288] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:52.678907 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56" is_pre_election: true
I20240502 20:45:52.679255 23133 raft_consensus.cc:2454] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 0.
I20240502 20:45:52.679773 23200 leader_election.cc:302] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:52.680049 24033 raft_consensus.cc:2781] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:52.680124 24033 raft_consensus.cc:482] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:52.680155 24033 raft_consensus.cc:3037] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:52.681053 24033 raft_consensus.cc:504] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:52.681463 24033 leader_election.cc:288] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Requested vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:52.682060 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56"
I20240502 20:45:52.682235 23133 raft_consensus.cc:3037] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:52.683099 23133 raft_consensus.cc:2454] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 1.
I20240502 20:45:52.683490 23200 leader_election.cc:302] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:52.683714 24033 raft_consensus.cc:2781] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:52.683815 24033 raft_consensus.cc:686] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Becoming Leader. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:45:52.684054 24033 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:52.686040 22985 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db3750c037e4393927784c55f66d349 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:52.784672 24033 consensus_queue.cc:1035] T b0f06fbb9f58417793a6d176b1f1078f P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:45:52.915467 24106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:52.915741 24106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:52.919323 24106 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 20:45:52.988920 24132 consensus_queue.cc:1035] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:53.025775 23133 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:53.026965 24033 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 101.244 ms
  time per row: 0.0506222 ms
I20240502 20:45:53.073909 22981 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41988:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:53.082157 23247 tablet_service.cc:1467] Processing CreateTablet for tablet 798923743c9d444f917b747309d2285c (DEFAULT_TABLE table=rebalance_test_table_2 [id=16f6b8a52dc94e42a53292a33382ae40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.082207 23113 tablet_service.cc:1467] Processing CreateTablet for tablet 798923743c9d444f917b747309d2285c (DEFAULT_TABLE table=rebalance_test_table_2 [id=16f6b8a52dc94e42a53292a33382ae40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.082616 23247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798923743c9d444f917b747309d2285c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.082753 23113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798923743c9d444f917b747309d2285c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.082733 23381 tablet_service.cc:1467] Processing CreateTablet for tablet df89654f87ed43758ba68a6a5582b059 (DEFAULT_TABLE table=rebalance_test_table_2 [id=16f6b8a52dc94e42a53292a33382ae40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.082734 23382 tablet_service.cc:1467] Processing CreateTablet for tablet 711786372ea5495abebec7086f958fef (DEFAULT_TABLE table=rebalance_test_table_2 [id=16f6b8a52dc94e42a53292a33382ae40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.083081 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df89654f87ed43758ba68a6a5582b059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.083242 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711786372ea5495abebec7086f958fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.082211 23246 tablet_service.cc:1467] Processing CreateTablet for tablet df89654f87ed43758ba68a6a5582b059 (DEFAULT_TABLE table=rebalance_test_table_2 [id=16f6b8a52dc94e42a53292a33382ae40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.086019 23110 tablet_service.cc:1467] Processing CreateTablet for tablet 711786372ea5495abebec7086f958fef (DEFAULT_TABLE table=rebalance_test_table_2 [id=16f6b8a52dc94e42a53292a33382ae40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.086099 23246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df89654f87ed43758ba68a6a5582b059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.086278 23110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711786372ea5495abebec7086f958fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.086915 24159 tablet_bootstrap.cc:492] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598: Bootstrap starting.
I20240502 20:45:53.086957 24158 tablet_bootstrap.cc:492] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349: Bootstrap starting.
I20240502 20:45:53.088320 24158 tablet_bootstrap.cc:654] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.088464 24159 tablet_bootstrap.cc:654] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.089702 24160 tablet_bootstrap.cc:492] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56: Bootstrap starting.
I20240502 20:45:53.089897 24159 tablet_bootstrap.cc:492] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598: No bootstrap required, opened a new log
I20240502 20:45:53.090010 24159 ts_tablet_manager.cc:1367] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:53.090471 24159 raft_consensus.cc:348] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.090580 24159 raft_consensus.cc:374] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.090620 24159 raft_consensus.cc:729] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.090780 24159 consensus_queue.cc:260] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.091028 24160 tablet_bootstrap.cc:654] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.091063 24159 ts_tablet_manager.cc:1398] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:53.091221 24159 tablet_bootstrap.cc:492] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598: Bootstrap starting.
I20240502 20:45:53.092180 24160 tablet_bootstrap.cc:492] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56: No bootstrap required, opened a new log
I20240502 20:45:53.092257 24160 ts_tablet_manager.cc:1367] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:53.092353 24159 tablet_bootstrap.cc:654] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.092700 24160 raft_consensus.cc:348] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.092797 24160 raft_consensus.cc:374] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.092825 24160 raft_consensus.cc:729] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.092944 24160 consensus_queue.cc:260] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.093250 24160 ts_tablet_manager.cc:1398] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:53.093430 24160 tablet_bootstrap.cc:492] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56: Bootstrap starting.
I20240502 20:45:53.094342 24160 tablet_bootstrap.cc:654] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.094765 24159 tablet_bootstrap.cc:492] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598: No bootstrap required, opened a new log
I20240502 20:45:53.094863 24159 ts_tablet_manager.cc:1367] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:53.095686 24159 raft_consensus.cc:348] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:53.095765 24158 tablet_bootstrap.cc:492] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349: No bootstrap required, opened a new log
I20240502 20:45:53.095826 24159 raft_consensus.cc:374] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.095867 24158 ts_tablet_manager.cc:1367] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.000s
I20240502 20:45:53.095868 24159 raft_consensus.cc:729] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.096022 24159 consensus_queue.cc:260] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:53.096366 24159 ts_tablet_manager.cc:1398] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:53.096354 24158 raft_consensus.cc:348] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.096457 24158 raft_consensus.cc:374] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.096485 24158 raft_consensus.cc:729] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.096606 24158 consensus_queue.cc:260] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.096887 24158 ts_tablet_manager.cc:1398] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:53.097095 24158 tablet_bootstrap.cc:492] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: Bootstrap starting.
I20240502 20:45:53.098135 24158 tablet_bootstrap.cc:654] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.099817 24160 tablet_bootstrap.cc:492] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56: No bootstrap required, opened a new log
I20240502 20:45:53.099920 24160 ts_tablet_manager.cc:1367] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 20:45:53.100416 24160 raft_consensus.cc:348] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:53.100519 24160 raft_consensus.cc:374] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.100565 24160 raft_consensus.cc:729] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.100699 24160 consensus_queue.cc:260] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:53.100952 24160 ts_tablet_manager.cc:1398] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:53.102898 24158 tablet_bootstrap.cc:492] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: No bootstrap required, opened a new log
I20240502 20:45:53.103006 24158 ts_tablet_manager.cc:1367] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:45:53.103528 24158 raft_consensus.cc:348] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.103633 24158 raft_consensus.cc:374] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.103662 24158 raft_consensus.cc:729] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.103780 24158 consensus_queue.cc:260] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.104058 24158 ts_tablet_manager.cc:1398] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:53.169683 24134 raft_consensus.cc:482] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:53.169890 24134 raft_consensus.cc:504] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.170527 24134 leader_election.cc:288] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:53.175839 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798923743c9d444f917b747309d2285c" candidate_uuid: "140dd1ea33be47e99530c773c5439d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db3750c037e4393927784c55f66d349" is_pre_election: true
I20240502 20:45:53.176123 23268 raft_consensus.cc:2454] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140dd1ea33be47e99530c773c5439d56 in term 0.
I20240502 20:45:53.176576 23064 leader_election.cc:302] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:53.176872 24134 raft_consensus.cc:2781] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:53.176995 24134 raft_consensus.cc:482] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:53.177104 24134 raft_consensus.cc:3037] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.178356 24134 raft_consensus.cc:504] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.178726 24134 leader_election.cc:288] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 election: Requested vote from peers 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:53.179477 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798923743c9d444f917b747309d2285c" candidate_uuid: "140dd1ea33be47e99530c773c5439d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db3750c037e4393927784c55f66d349"
I20240502 20:45:53.179631 23268 raft_consensus.cc:3037] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.180970 23268 raft_consensus.cc:2454] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140dd1ea33be47e99530c773c5439d56 in term 1.
I20240502 20:45:53.181599 23064 leader_election.cc:302] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:53.181877 24134 raft_consensus.cc:2781] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:53.181978 24134 raft_consensus.cc:686] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Becoming Leader. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Running, Role: LEADER
I20240502 20:45:53.182147 24134 consensus_queue.cc:237] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.184091 22980 catalog_manager.cc:5484] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 140dd1ea33be47e99530c773c5439d56 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:53.235116 24038 raft_consensus.cc:482] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:53.235292 24038 raft_consensus.cc:504] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.235735 24038 leader_election.cc:288] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:53.236423 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df89654f87ed43758ba68a6a5582b059" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598" is_pre_election: true
I20240502 20:45:53.236582 23404 raft_consensus.cc:2454] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 0.
I20240502 20:45:53.236982 23199 leader_election.cc:302] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:53.237195 24038 raft_consensus.cc:2781] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:53.237267 24038 raft_consensus.cc:482] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:53.237304 24038 raft_consensus.cc:3037] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.238262 24038 raft_consensus.cc:504] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.238579 24038 leader_election.cc:288] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Requested vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:53.239089 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df89654f87ed43758ba68a6a5582b059" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
I20240502 20:45:53.239214 23404 raft_consensus.cc:3037] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.240028 23404 raft_consensus.cc:2454] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 1.
I20240502 20:45:53.240384 23199 leader_election.cc:302] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:53.240576 24038 raft_consensus.cc:2781] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:53.240690 24038 raft_consensus.cc:686] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Becoming Leader. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:45:53.240854 24038 consensus_queue.cc:237] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.242842 22980 catalog_manager.cc:5484] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db3750c037e4393927784c55f66d349 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:53.327769 24032 raft_consensus.cc:482] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:53.327932 24032 raft_consensus.cc:504] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:53.328316 24032 leader_election.cc:288] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:53.328958 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711786372ea5495abebec7086f958fef" candidate_uuid: "ee6fe9bcf23a450cb5432a8205660598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56" is_pre_election: true
I20240502 20:45:53.329167 23133 raft_consensus.cc:2454] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee6fe9bcf23a450cb5432a8205660598 in term 0.
I20240502 20:45:53.329705 23334 leader_election.cc:302] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:53.329914 24032 raft_consensus.cc:2781] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:53.329985 24032 raft_consensus.cc:482] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:53.330155 24032 raft_consensus.cc:3037] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.331116 24032 raft_consensus.cc:504] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:53.331440 24032 leader_election.cc:288] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 election: Requested vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:53.331965 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711786372ea5495abebec7086f958fef" candidate_uuid: "ee6fe9bcf23a450cb5432a8205660598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56"
I20240502 20:45:53.332126 23133 raft_consensus.cc:3037] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.333065 23133 raft_consensus.cc:2454] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee6fe9bcf23a450cb5432a8205660598 in term 1.
I20240502 20:45:53.333463 23334 leader_election.cc:302] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:53.333900 24032 raft_consensus.cc:2781] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:53.334308 24032 raft_consensus.cc:686] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Becoming Leader. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Running, Role: LEADER
I20240502 20:45:53.334556 24032 consensus_queue.cc:237] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } }
I20240502 20:45:53.336601 22980 catalog_manager.cc:5484] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:53.535164 24172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:53.535491 24172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:53.539049 24172 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 20:45:53.574422 24134 consensus_queue.cc:1035] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:53.650435 23404 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:53.650935 24038 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:53.654511 23133 raft_consensus.cc:1261] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:53.655906 24032 consensus_queue.cc:1035] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:53.667299 24211 mvcc.cc:205] Tried to move back new op lower bound from 7023340558941392896 to 7023340557275725824. Current Snapshot: MvccSnapshot[applied={T|T < 7023340558941392896}]

INSERT report
    rows total: 2000
    time total: 112.867 ms
  time per row: 0.0564336 ms
I20240502 20:45:53.711366 22978 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41988:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:53.718092 23110 tablet_service.cc:1467] Processing CreateTablet for tablet be76bba97022462a86ed4a4cc2a2dd64 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d084e24e8342d892514572e869edd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.718492 23110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be76bba97022462a86ed4a4cc2a2dd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.718850 23381 tablet_service.cc:1467] Processing CreateTablet for tablet 98e34e4543d24b46963049828877d3d5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d084e24e8342d892514572e869edd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.719015 23382 tablet_service.cc:1467] Processing CreateTablet for tablet 1e226ee7a67b46f687971e7de8099217 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d084e24e8342d892514572e869edd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.719169 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e34e4543d24b46963049828877d3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.719368 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e226ee7a67b46f687971e7de8099217. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.719550 23246 tablet_service.cc:1467] Processing CreateTablet for tablet 1e226ee7a67b46f687971e7de8099217 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d084e24e8342d892514572e869edd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.719872 23246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e226ee7a67b46f687971e7de8099217. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.721576 23247 tablet_service.cc:1467] Processing CreateTablet for tablet be76bba97022462a86ed4a4cc2a2dd64 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d084e24e8342d892514572e869edd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.721871 23247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be76bba97022462a86ed4a4cc2a2dd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.722651 24228 tablet_bootstrap.cc:492] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598: Bootstrap starting.
I20240502 20:45:53.723839 24227 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: Bootstrap starting.
I20240502 20:45:53.723944 24228 tablet_bootstrap.cc:654] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.724594 23110 tablet_service.cc:1467] Processing CreateTablet for tablet 98e34e4543d24b46963049828877d3d5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6d084e24e8342d892514572e869edd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:53.724867 24227 tablet_bootstrap.cc:654] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.724922 23110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e34e4543d24b46963049828877d3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:53.725894 24228 tablet_bootstrap.cc:492] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598: No bootstrap required, opened a new log
I20240502 20:45:53.726055 24228 ts_tablet_manager.cc:1367] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:53.726881 24228 raft_consensus.cc:348] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.727041 24228 raft_consensus.cc:374] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.727077 24227 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: No bootstrap required, opened a new log
I20240502 20:45:53.727087 24228 raft_consensus.cc:729] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.727205 24227 ts_tablet_manager.cc:1367] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:53.727502 24229 tablet_bootstrap.cc:492] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349: Bootstrap starting.
I20240502 20:45:53.727656 24228 consensus_queue.cc:260] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.727909 24227 raft_consensus.cc:348] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.728276 24227 raft_consensus.cc:374] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.728425 24227 raft_consensus.cc:729] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.728430 24228 ts_tablet_manager.cc:1398] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 20:45:53.728649 24228 tablet_bootstrap.cc:492] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598: Bootstrap starting.
I20240502 20:45:53.728691 24227 consensus_queue.cc:260] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.728936 24229 tablet_bootstrap.cc:654] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.729552 24227 ts_tablet_manager.cc:1398] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:53.729835 24228 tablet_bootstrap.cc:654] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.729861 24227 tablet_bootstrap.cc:492] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56: Bootstrap starting.
I20240502 20:45:53.731279 24229 tablet_bootstrap.cc:492] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349: No bootstrap required, opened a new log
I20240502 20:45:53.731372 24228 tablet_bootstrap.cc:492] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598: No bootstrap required, opened a new log
I20240502 20:45:53.731411 24229 ts_tablet_manager.cc:1367] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:45:53.731456 24228 ts_tablet_manager.cc:1367] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:45:53.731940 24229 raft_consensus.cc:348] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.732057 24229 raft_consensus.cc:374] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.731976 24228 raft_consensus.cc:348] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.732102 24229 raft_consensus.cc:729] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.732131 24228 raft_consensus.cc:374] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.732167 24228 raft_consensus.cc:729] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.732247 24229 consensus_queue.cc:260] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.732291 24228 consensus_queue.cc:260] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.732560 24229 ts_tablet_manager.cc:1398] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:53.732582 24228 ts_tablet_manager.cc:1398] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.001s
I20240502 20:45:53.732766 24229 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: Bootstrap starting.
I20240502 20:45:53.733407 24227 tablet_bootstrap.cc:654] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.733790 24229 tablet_bootstrap.cc:654] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:53.735430 24229 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: No bootstrap required, opened a new log
I20240502 20:45:53.735540 24229 ts_tablet_manager.cc:1367] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:53.736183 24229 raft_consensus.cc:348] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.736284 24229 raft_consensus.cc:374] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.736306 24229 raft_consensus.cc:729] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.736331 24227 tablet_bootstrap.cc:492] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56: No bootstrap required, opened a new log
I20240502 20:45:53.736450 24227 ts_tablet_manager.cc:1367] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 20:45:53.736424 24229 consensus_queue.cc:260] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:53.736725 24229 ts_tablet_manager.cc:1398] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:53.736989 24227 raft_consensus.cc:348] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.737195 24227 raft_consensus.cc:374] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:53.737251 24227 raft_consensus.cc:729] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Initialized, Role: FOLLOWER
I20240502 20:45:53.737425 24227 consensus_queue.cc:260] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.737756 24227 ts_tablet_manager.cc:1398] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:53.898672 24033 raft_consensus.cc:482] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:53.898844 24033 raft_consensus.cc:504] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.899268 24033 leader_election.cc:288] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:53.899940 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e226ee7a67b46f687971e7de8099217" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598" is_pre_election: true
I20240502 20:45:53.900101 23404 raft_consensus.cc:2454] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 0.
I20240502 20:45:53.900471 23199 leader_election.cc:302] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:53.900804 24033 raft_consensus.cc:2781] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:53.900918 24033 raft_consensus.cc:482] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:53.900950 24033 raft_consensus.cc:3037] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.902036 24033 raft_consensus.cc:504] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.902380 24033 leader_election.cc:288] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Requested vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:53.902894 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e226ee7a67b46f687971e7de8099217" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
I20240502 20:45:53.903016 23404 raft_consensus.cc:3037] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.903831 23404 raft_consensus.cc:2454] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 1.
I20240502 20:45:53.904177 23199 leader_election.cc:302] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:53.904422 24033 raft_consensus.cc:2781] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:53.904525 24033 raft_consensus.cc:686] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Becoming Leader. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:45:53.904703 24033 consensus_queue.cc:237] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.906764 22978 catalog_manager.cc:5484] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db3750c037e4393927784c55f66d349 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:53.965037 24132 raft_consensus.cc:482] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:53.965248 24132 raft_consensus.cc:504] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.965662 24132 leader_election.cc:288] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:53.966285 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e34e4543d24b46963049828877d3d5" candidate_uuid: "140dd1ea33be47e99530c773c5439d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598" is_pre_election: true
I20240502 20:45:53.966449 23404 raft_consensus.cc:2454] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140dd1ea33be47e99530c773c5439d56 in term 0.
I20240502 20:45:53.966868 23063 leader_election.cc:302] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:53.967061 24132 raft_consensus.cc:2781] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:53.967120 24132 raft_consensus.cc:482] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:53.967170 24132 raft_consensus.cc:3037] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.968386 24132 raft_consensus.cc:504] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.968910 24132 leader_election.cc:288] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 election: Requested vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:53.969583 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e34e4543d24b46963049828877d3d5" candidate_uuid: "140dd1ea33be47e99530c773c5439d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
I20240502 20:45:53.969750 23404 raft_consensus.cc:3037] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:53.970631 23404 raft_consensus.cc:2454] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140dd1ea33be47e99530c773c5439d56 in term 1.
I20240502 20:45:53.971091 23063 leader_election.cc:302] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:53.971330 24132 raft_consensus.cc:2781] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:53.971424 24132 raft_consensus.cc:686] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Becoming Leader. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Running, Role: LEADER
I20240502 20:45:53.971582 24132 consensus_queue.cc:237] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:53.973505 22978 catalog_manager.cc:5484] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 140dd1ea33be47e99530c773c5439d56 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:54.035619 24132 raft_consensus.cc:482] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:54.035765 24132 raft_consensus.cc:504] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:54.036159 24132 leader_election.cc:288] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:54.037168 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64" candidate_uuid: "140dd1ea33be47e99530c773c5439d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db3750c037e4393927784c55f66d349" is_pre_election: true
I20240502 20:45:54.037439 23268 raft_consensus.cc:2454] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140dd1ea33be47e99530c773c5439d56 in term 0.
I20240502 20:45:54.037935 23064 leader_election.cc:302] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:54.038228 24132 raft_consensus.cc:2781] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:54.038321 24132 raft_consensus.cc:482] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:54.038352 24132 raft_consensus.cc:3037] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.039667 24132 raft_consensus.cc:504] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:54.040098 24132 leader_election.cc:288] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 election: Requested vote from peers 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:54.040884 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64" candidate_uuid: "140dd1ea33be47e99530c773c5439d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db3750c037e4393927784c55f66d349"
I20240502 20:45:54.041117 23268 raft_consensus.cc:3037] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.042363 23268 raft_consensus.cc:2454] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140dd1ea33be47e99530c773c5439d56 in term 1.
I20240502 20:45:54.042910 23064 leader_election.cc:302] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:54.043309 24132 raft_consensus.cc:2781] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:54.043442 24132 raft_consensus.cc:686] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Becoming Leader. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Running, Role: LEADER
I20240502 20:45:54.043615 24132 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:54.045827 22978 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 140dd1ea33be47e99530c773c5439d56 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:54.219442 24239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:54.219808 24239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:54.223235 24239 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 20:45:54.332032 23404 raft_consensus.cc:1261] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.332562 24132 consensus_queue.cc:1035] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.333503 23404 raft_consensus.cc:1261] T 1e226ee7a67b46f687971e7de8099217 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.334062 24033 consensus_queue.cc:1035] T 1e226ee7a67b46f687971e7de8099217 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.341369 23268 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.342031 24134 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 113.983 ms
  time per row: 0.0569914 ms
I20240502 20:45:54.388813 22978 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41988:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:45:54.397006 23246 tablet_service.cc:1467] Processing CreateTablet for tablet 21b3fcb144894a818b2f649a4473a7ee (DEFAULT_TABLE table=rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.397245 23110 tablet_service.cc:1467] Processing CreateTablet for tablet 21b3fcb144894a818b2f649a4473a7ee (DEFAULT_TABLE table=rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.397523 23110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b3fcb144894a818b2f649a4473a7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.397518 23246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b3fcb144894a818b2f649a4473a7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.397441 23381 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5fdd7816e345038ed9b6946084a67e (DEFAULT_TABLE table=rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.397723 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5fdd7816e345038ed9b6946084a67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.400238 24291 tablet_bootstrap.cc:492] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349: Bootstrap starting.
I20240502 20:45:54.400223 23382 tablet_service.cc:1467] Processing CreateTablet for tablet 4b80c0d7e5de4ad58511694ec615dfef (DEFAULT_TABLE table=rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.400223 23113 tablet_service.cc:1467] Processing CreateTablet for tablet 4b80c0d7e5de4ad58511694ec615dfef (DEFAULT_TABLE table=rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.400485 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b80c0d7e5de4ad58511694ec615dfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.400485 23113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b80c0d7e5de4ad58511694ec615dfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.401022 24291 tablet_bootstrap.cc:654] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.401724 24292 tablet_bootstrap.cc:492] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56: Bootstrap starting.
I20240502 20:45:54.402459 24292 tablet_bootstrap.cc:654] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.397053 23247 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5fdd7816e345038ed9b6946084a67e (DEFAULT_TABLE table=rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:54.403059 24290 tablet_bootstrap.cc:492] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: Bootstrap starting.
I20240502 20:45:54.403178 23247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5fdd7816e345038ed9b6946084a67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:54.403381 24291 tablet_bootstrap.cc:492] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349: No bootstrap required, opened a new log
I20240502 20:45:54.403451 24291 ts_tablet_manager.cc:1367] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:45:54.403823 24291 raft_consensus.cc:348] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:54.403910 24291 raft_consensus.cc:374] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.403931 24291 raft_consensus.cc:729] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.403939 24290 tablet_bootstrap.cc:654] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.404048 24291 consensus_queue.cc:260] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:54.404309 24291 ts_tablet_manager.cc:1398] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:54.405315 24290 tablet_bootstrap.cc:492] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: No bootstrap required, opened a new log
I20240502 20:45:54.405408 24290 ts_tablet_manager.cc:1367] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:54.405776 24292 tablet_bootstrap.cc:492] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56: No bootstrap required, opened a new log
I20240502 20:45:54.405841 24291 tablet_bootstrap.cc:492] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349: Bootstrap starting.
I20240502 20:45:54.405866 24292 ts_tablet_manager.cc:1367] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.000s
I20240502 20:45:54.405925 24290 raft_consensus.cc:348] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.406025 24290 raft_consensus.cc:374] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.406158 24290 raft_consensus.cc:729] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.406329 24290 consensus_queue.cc:260] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.406354 24292 raft_consensus.cc:348] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:54.406442 24292 raft_consensus.cc:374] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.406466 24292 raft_consensus.cc:729] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.406634 24290 ts_tablet_manager.cc:1398] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:45:54.406632 24292 consensus_queue.cc:260] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:54.406798 24290 tablet_bootstrap.cc:492] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: Bootstrap starting.
I20240502 20:45:54.407168 24292 ts_tablet_manager.cc:1398] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:54.407490 24292 tablet_bootstrap.cc:492] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: Bootstrap starting.
I20240502 20:45:54.408118 24290 tablet_bootstrap.cc:654] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.408689 24291 tablet_bootstrap.cc:654] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.409519 24292 tablet_bootstrap.cc:654] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:54.409765 24290 tablet_bootstrap.cc:492] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: No bootstrap required, opened a new log
I20240502 20:45:54.409837 24291 tablet_bootstrap.cc:492] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349: No bootstrap required, opened a new log
I20240502 20:45:54.409854 24290 ts_tablet_manager.cc:1367] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:54.409916 24291 ts_tablet_manager.cc:1367] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 20:45:54.410380 24290 raft_consensus.cc:348] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.410380 24291 raft_consensus.cc:348] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.410499 24290 raft_consensus.cc:374] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.410511 24291 raft_consensus.cc:374] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.410540 24290 raft_consensus.cc:729] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.410547 24291 raft_consensus.cc:729] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.410678 24290 consensus_queue.cc:260] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.410678 24291 consensus_queue.cc:260] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.410964 24290 ts_tablet_manager.cc:1398] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:54.410964 24291 ts_tablet_manager.cc:1398] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:45:54.413002 24292 tablet_bootstrap.cc:492] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: No bootstrap required, opened a new log
I20240502 20:45:54.413166 24292 ts_tablet_manager.cc:1367] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.001s
I20240502 20:45:54.414885 24292 raft_consensus.cc:348] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.415264 24292 raft_consensus.cc:374] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:54.415318 24292 raft_consensus.cc:729] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Initialized, Role: FOLLOWER
I20240502 20:45:54.415518 24292 consensus_queue.cc:260] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.415884 24292 ts_tablet_manager.cc:1398] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240502 20:45:54.450065 24276 raft_consensus.cc:482] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:54.450254 24276 raft_consensus.cc:504] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.450680 24276 leader_election.cc:288] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:54.451185 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b80c0d7e5de4ad58511694ec615dfef" candidate_uuid: "ee6fe9bcf23a450cb5432a8205660598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56" is_pre_election: true
I20240502 20:45:54.451344 23133 raft_consensus.cc:2454] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee6fe9bcf23a450cb5432a8205660598 in term 0.
I20240502 20:45:54.451639 23334 leader_election.cc:302] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:54.451817 24276 raft_consensus.cc:2781] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:54.451879 24276 raft_consensus.cc:482] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:54.451907 24276 raft_consensus.cc:3037] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.452838 24276 raft_consensus.cc:504] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.453178 24276 leader_election.cc:288] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 election: Requested vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:54.453630 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b80c0d7e5de4ad58511694ec615dfef" candidate_uuid: "ee6fe9bcf23a450cb5432a8205660598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56"
I20240502 20:45:54.453764 23133 raft_consensus.cc:3037] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.454583 23133 raft_consensus.cc:2454] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee6fe9bcf23a450cb5432a8205660598 in term 1.
I20240502 20:45:54.454908 23334 leader_election.cc:302] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:54.455076 24276 raft_consensus.cc:2781] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:54.455241 24276 raft_consensus.cc:686] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Becoming Leader. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Running, Role: LEADER
I20240502 20:45:54.455432 24276 consensus_queue.cc:237] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.457342 22978 catalog_manager.cc:5484] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:54.470103 24276 raft_consensus.cc:482] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:54.470283 24276 raft_consensus.cc:504] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.470821 24276 leader_election.cc:288] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:54.475383 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5fdd7816e345038ed9b6946084a67e" candidate_uuid: "ee6fe9bcf23a450cb5432a8205660598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db3750c037e4393927784c55f66d349" is_pre_election: true
I20240502 20:45:54.475574 23268 raft_consensus.cc:2454] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee6fe9bcf23a450cb5432a8205660598 in term 0.
I20240502 20:45:54.475950 23334 leader_election.cc:302] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:54.476218 24276 raft_consensus.cc:2781] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:54.476284 24276 raft_consensus.cc:482] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:54.476349 24276 raft_consensus.cc:3037] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.477324 24276 raft_consensus.cc:504] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.477645 24276 leader_election.cc:288] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 election: Requested vote from peers 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:54.478197 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5fdd7816e345038ed9b6946084a67e" candidate_uuid: "ee6fe9bcf23a450cb5432a8205660598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db3750c037e4393927784c55f66d349"
I20240502 20:45:54.478338 23268 raft_consensus.cc:3037] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.479207 23268 raft_consensus.cc:2454] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee6fe9bcf23a450cb5432a8205660598 in term 1.
I20240502 20:45:54.479532 23334 leader_election.cc:302] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:54.479764 24276 raft_consensus.cc:2781] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:54.479897 24276 raft_consensus.cc:686] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Becoming Leader. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Running, Role: LEADER
I20240502 20:45:54.480037 24276 consensus_queue.cc:237] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } }
I20240502 20:45:54.482138 22978 catalog_manager.cc:5484] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:54.545017 24038 raft_consensus.cc:482] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:54.545265 24038 raft_consensus.cc:504] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:54.545786 24038 leader_election.cc:288] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:54.546440 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b3fcb144894a818b2f649a4473a7ee" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56" is_pre_election: true
I20240502 20:45:54.546607 23133 raft_consensus.cc:2454] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 0.
I20240502 20:45:54.546936 23200 leader_election.cc:302] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:54.547129 24038 raft_consensus.cc:2781] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:45:54.547196 24038 raft_consensus.cc:482] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:54.547235 24038 raft_consensus.cc:3037] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.548141 24038 raft_consensus.cc:504] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:54.548540 24038 leader_election.cc:288] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Requested vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:54.549091 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21b3fcb144894a818b2f649a4473a7ee" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "140dd1ea33be47e99530c773c5439d56"
I20240502 20:45:54.549234 23133 raft_consensus.cc:3037] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:54.550078 23133 raft_consensus.cc:2454] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 1.
I20240502 20:45:54.550441 23200 leader_election.cc:302] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 140dd1ea33be47e99530c773c5439d56, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:54.550643 24038 raft_consensus.cc:2781] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:54.550738 24038 raft_consensus.cc:686] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Becoming Leader. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:45:54.550897 24038 consensus_queue.cc:237] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } }
I20240502 20:45:54.552891 22978 catalog_manager.cc:5484] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db3750c037e4393927784c55f66d349 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } }
W20240502 20:45:54.699978 24303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:54.700336 24303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:54.704205 24303 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 20:45:54.803359 23268 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.804018 24276 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.805085 23133 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.805686 24276 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:54.812515 23133 raft_consensus.cc:1261] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:45:54.814188 24038 consensus_queue.cc:1035] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 101.577 ms
  time per row: 0.0507885 ms
I20240502 20:45:54.861179 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.132:45001
--local_ip_for_outbound_sockets=127.19.66.132
--tserver_master_addrs=127.19.66.190:46265
--webserver_port=37929
--webserver_interface=127.19.66.132
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:45:55.002923 24336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:55.003216 24336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:55.003268 24336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:55.006884 24336 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:55.006985 24336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:55.007077 24336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.132
I20240502 20:45:55.011433 24336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.132:45001
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.66.132
--webserver_port=37929
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.24336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.132
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:55.012576 24336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:55.013882 24336 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:55.024847 24341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:55.025493 24344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:55.026129 24342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:55.026895 24336 server_base.cc:1034] running on GCE node
I20240502 20:45:55.027443 24336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:55.028158 24336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:55.029366 24336 hybrid_clock.cc:648] HybridClock initialized: now 1714682755029345 us; error 33 us; skew 500 ppm
I20240502 20:45:55.031677 24336 webserver.cc:460] Webserver started at http://127.19.66.132:37929/ using document root <none> and password file <none>
I20240502 20:45:55.032327 24336 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:55.032408 24336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:55.036615 24336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:55.039515 24350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:55.041136 24336 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:55.041292 24336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "94ac80c7dd7f4767bb357b89c547d8b2"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:55.041821 24336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:55.078562 24336 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:55.079349 24336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:55.079530 24336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:55.080179 24336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:55.081573 24358 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 20:45:55.087754 24336 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 20:45:55.087863 24336 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.000s
I20240502 20:45:55.087953 24336 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 20:45:55.089877 24336 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 20:45:55.089944 24336 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:55.090289 24358 tablet_bootstrap.cc:492] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap starting.
I20240502 20:45:55.102576 24336 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.132:45001
I20240502 20:45:55.102844 24467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.132:45001 every 8 connection(s)
I20240502 20:45:55.103935 24336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:45:55.107676 24358 log.cc:824] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:55.109416 24358 tablet_bootstrap.cc:492] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:55.109812 24358 tablet_bootstrap.cc:492] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap complete.
I20240502 20:45:55.110782 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 24336
I20240502 20:45:55.111279 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.133:45729
--local_ip_for_outbound_sockets=127.19.66.133
--tserver_master_addrs=127.19.66.190:46265
--webserver_port=45695
--webserver_interface=127.19.66.133
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:55.111294 24358 ts_tablet_manager.cc:1367] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.010s
I20240502 20:45:55.115052 24358 raft_consensus.cc:348] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:55.115509 24358 raft_consensus.cc:729] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ac80c7dd7f4767bb357b89c547d8b2, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.116231 24468 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:55.116197 24358 consensus_queue.cc:260] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:55.116570 24468 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:55.117419 24358 ts_tablet_manager.cc:1398] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20240502 20:45:55.117542 24468 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:55.120545 22979 ts_manager.cc:194] Re-registered known tserver with Master: 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132:45001)
I20240502 20:45:55.121938 22979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.132:45531
I20240502 20:45:55.125211 24468 heartbeater.cc:503] Master 127.19.66.190:46265 was elected leader, sending a full tablet report...
W20240502 20:45:55.249990 24472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:55.250252 24472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:55.250300 24472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:55.254938 24472 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:55.255052 24472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:55.255139 24472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.133
I20240502 20:45:55.260984 24472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.133:45729
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.66.133
--webserver_port=45695
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.24472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.133
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:55.262221 24472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:55.263576 24472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:55.275353 24479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:55.275780 24481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:55.276279 24478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:55.277117 24472 server_base.cc:1034] running on GCE node
I20240502 20:45:55.277704 24472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:55.278414 24472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:55.279636 24472 hybrid_clock.cc:648] HybridClock initialized: now 1714682755279624 us; error 39 us; skew 500 ppm
I20240502 20:45:55.282107 24472 webserver.cc:460] Webserver started at http://127.19.66.133:45695/ using document root <none> and password file <none>
I20240502 20:45:55.282765 24472 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:55.282860 24472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:55.288775 24472 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 20:45:55.291836 24487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:55.293164 24472 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 20:45:55.293313 24472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b30503059718458bb016185550f5e0fb"
format_stamp: "Formatted at 2024-05-02 20:45:46 on dist-test-slave-gqqg"
I20240502 20:45:55.293859 24472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:55.307360 24472 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:55.308106 24472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:55.308301 24472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:55.308923 24472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:55.310094 24472 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:55.310149 24472 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:55.310201 24472 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:55.310248 24472 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:55.321120 24472 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.133:45729
I20240502 20:45:55.321605 24603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.133:45729 every 8 connection(s)
I20240502 20:45:55.322553 24472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:45:55.331326 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 24472
I20240502 20:45:55.331831 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.134:43861
--local_ip_for_outbound_sockets=127.19.66.134
--tserver_master_addrs=127.19.66.190:46265
--webserver_port=43609
--webserver_interface=127.19.66.134
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:55.333349 24604 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:55.333624 24604 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:55.334343 24604 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:55.335625 22979 ts_manager.cc:194] Re-registered known tserver with Master: b30503059718458bb016185550f5e0fb (127.19.66.133:45729)
I20240502 20:45:55.336386 22979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.133:44691
W20240502 20:45:55.355041 24469 tablet.cc:2369] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:45:55.472038 24607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:55.472321 24607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:55.472379 24607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:55.476203 24607 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:55.476292 24607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:55.476368 24607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.134
I20240502 20:45:55.480643 24607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.134:43861
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.19.66.134
--webserver_port=43609
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.24607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.134
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:55.481834 24607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:55.483189 24607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:55.494747 24612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:55.495070 24615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:55.495808 24613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:55.496675 24607 server_base.cc:1034] running on GCE node
I20240502 20:45:55.497447 24607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:55.498518 24607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:55.499743 24607 hybrid_clock.cc:648] HybridClock initialized: now 1714682755499716 us; error 50 us; skew 500 ppm
I20240502 20:45:55.502121 24607 webserver.cc:460] Webserver started at http://127.19.66.134:43609/ using document root <none> and password file <none>
I20240502 20:45:55.502887 24607 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:55.502986 24607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:55.507097 24607 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 20:45:55.509378 24621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:55.510615 24607 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 20:45:55.510744 24607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c9af0b94bfe2492bb34c7255879a7328"
format_stamp: "Formatted at 2024-05-02 20:45:47 on dist-test-slave-gqqg"
I20240502 20:45:55.511267 24607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:55.538060 24607 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:55.538959 24607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:55.539184 24607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:55.539906 24607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:55.541229 24607 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:55.541306 24607 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:55.541373 24607 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:55.541407 24607 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:55.553776 24607 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.134:43861
I20240502 20:45:55.553802 24735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.134:43861 every 8 connection(s)
I20240502 20:45:55.555001 24607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:45:55.560366 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 24607
I20240502 20:45:55.560798 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.135:35917
--local_ip_for_outbound_sockets=127.19.66.135
--tserver_master_addrs=127.19.66.190:46265
--webserver_port=45889
--webserver_interface=127.19.66.135
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:45:55.565685 24736 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:55.566164 24736 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:55.567159 24736 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:55.568948 22979 ts_manager.cc:194] Re-registered known tserver with Master: c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861)
I20240502 20:45:55.569932 22979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.134:41593
W20240502 20:45:55.691336 24739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:45:55.691587 24739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:45:55.691671 24739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:45:55.695189 24739 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:45:55.695297 24739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:45:55.695407 24739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.135
I20240502 20:45:55.700057 24739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.135:35917
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.19.66.135
--webserver_port=45889
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.24739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.135
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:45:55.701262 24739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:55.702723 24739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:55.713228 24745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:55.713855 24744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:55.714450 24747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:55.715284 24739 server_base.cc:1034] running on GCE node
I20240502 20:45:55.715984 24739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:45:55.717103 24739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:45:55.718396 24739 hybrid_clock.cc:648] HybridClock initialized: now 1714682755718364 us; error 70 us; skew 500 ppm
I20240502 20:45:55.720853 24739 webserver.cc:460] Webserver started at http://127.19.66.135:45889/ using document root <none> and password file <none>
I20240502 20:45:55.721544 24739 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:55.721633 24739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:55.726306 24739 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 20:45:55.729123 24753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:55.730993 24739 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:45:55.731237 24739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "09f2f86617004c96b1e186759506cc68"
format_stamp: "Formatted at 2024-05-02 20:45:47 on dist-test-slave-gqqg"
I20240502 20:45:55.731879 24739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:55.752698 24739 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:55.753544 24739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:55.753739 24739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:55.754495 24739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:55.755823 24760 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 20:45:55.760159 24739 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 20:45:55.760254 24739 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s	user 0.001s	sys 0.000s
I20240502 20:45:55.760356 24739 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 20:45:55.762418 24739 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 20:45:55.762491 24739 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:55.762812 24760 tablet_bootstrap.cc:492] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Bootstrap starting.
I20240502 20:45:55.777192 24739 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.135:35917
I20240502 20:45:55.778435 24739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:45:55.779582 24869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.135:35917 every 8 connection(s)
I20240502 20:45:55.780048 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 24739
I20240502 20:45:55.780079 24760 log.cc:824] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:55.780244 19722 test_util.cc:268] Using random seed: -1332769132
I20240502 20:45:55.783480 24760 tablet_bootstrap.cc:492] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:45:55.783933 24760 tablet_bootstrap.cc:492] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Bootstrap complete.
I20240502 20:45:55.785241 24760 ts_tablet_manager.cc:1367] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Time spent bootstrapping tablet: real 0.023s	user 0.008s	sys 0.014s
I20240502 20:45:55.789808 24760 raft_consensus.cc:348] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:55.790544 24760 raft_consensus.cc:729] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Initialized, Role: FOLLOWER
I20240502 20:45:55.791177 24760 consensus_queue.cc:260] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:55.792407 24760 ts_tablet_manager.cc:1398] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20240502 20:45:55.794293 19722 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 20:45:55.794574 19722 test_util.cc:268] Using random seed: -1332754804
I20240502 20:45:55.794965 24870 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:45:55.795228 24870 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:55.796059 24870 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:45:55.799784 22979 ts_manager.cc:194] Re-registered known tserver with Master: 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917)
I20240502 20:45:55.801113 22979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.135:42509
I20240502 20:45:55.804466 24870 heartbeater.cc:503] Master 127.19.66.190:46265 was elected leader, sending a full tablet report...
I20240502 20:45:55.809468 19722 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 20:45:55.809710 19722 test_util.cc:268] Using random seed: -1332739668
I20240502 20:45:55.827594 19722 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 20:45:55.827811 19722 test_util.cc:268] Using random seed: -1332721566
I20240502 20:45:55.846530 19722 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 20:45:55.846814 19722 test_util.cc:268] Using random seed: -1332702567
I20240502 20:45:55.859830 19722 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240502 20:45:56.029743 24871 tablet.cc:2369] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:45:56.339272 24604 heartbeater.cc:503] Master 127.19.66.190:46265 was elected leader, sending a full tablet report...
I20240502 20:45:56.376202 23113 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:56.377751 24790 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:56.380025 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:56.395001 23247 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:45:56.401487 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:56.406941 24536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:56.423538 23382 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:56.573554 24736 heartbeater.cc:503] Master 127.19.66.190:46265 was elected leader, sending a full tablet report...
I20240502 20:45:56.726758 24790 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:56.737813 25059 raft_consensus.cc:482] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:56.738356 25059 raft_consensus.cc:504] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:56.746376 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:56.747416 25059 leader_election.cc:288] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917)
I20240502 20:45:56.761695 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:56.768254 24824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d54cbc40bcc4fa481a04d100ed3c927" candidate_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f2f86617004c96b1e186759506cc68" is_pre_election: true
I20240502 20:45:56.768654 24824 raft_consensus.cc:2396] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94ac80c7dd7f4767bb357b89c547d8b2 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 20:45:56.771114 24351 leader_election.cc:302] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 94ac80c7dd7f4767bb357b89c547d8b2; no voters: 09f2f86617004c96b1e186759506cc68
I20240502 20:45:56.771659 23382 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:56.775480 24536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:45:56.777259 23113 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:45:56.785432 25059 raft_consensus.cc:2726] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:45:56.838265 23247 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:45:56.945338 23402 consensus_queue.cc:237] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:45:56.953739 23131 raft_consensus.cc:1261] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240502 20:45:56.956231 24995 consensus_queue.cc:1035] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
W20240502 20:45:56.956959 23333 consensus_peers.cc:485] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 -> Peer b30503059718458bb016185550f5e0fb (127.19.66.133:45729): Couldn't send request to peer b30503059718458bb016185550f5e0fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 711786372ea5495abebec7086f958fef. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:56.964888 24995 raft_consensus.cc:2932] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index -1 to 33, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } } }
I20240502 20:45:56.969275 22978 catalog_manager.cc:5484] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: config changed from index -1 to 33, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:56.979687 23131 raft_consensus.cc:2932] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index -1 to 33, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } } }
I20240502 20:45:56.996660 23402 consensus_queue.cc:237] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:56.998683 23267 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20240502 20:45:56.999603 24995 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 20:45:57.003479 24995 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index -1 to 34, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.004662 23266 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index -1 to 34, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
W20240502 20:45:57.006556 23336 consensus_peers.cc:485] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 -> Peer c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861): Couldn't send request to peer c9af0b94bfe2492bb34c7255879a7328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a5fdd7816e345038ed9b6946084a67e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.009163 22978 catalog_manager.cc:5484] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index -1 to 34, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.066299 23131 consensus_queue.cc:237] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } }
I20240502 20:45:57.070235 23404 raft_consensus.cc:1261] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20240502 20:45:57.072474 24967 consensus_queue.cc:1035] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 34, Time since last communication: 0.000s
W20240502 20:45:57.077910 23066 consensus_peers.cc:485] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 -> Peer 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132:45001): Couldn't send request to peer 94ac80c7dd7f4767bb357b89c547d8b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98e34e4543d24b46963049828877d3d5. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.091768 24134 raft_consensus.cc:2932] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index -1 to 36, NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } } }
I20240502 20:45:57.098587 22978 catalog_manager.cc:5484] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: config changed from index -1 to 36, NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) added. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:57.108337 23404 raft_consensus.cc:2932] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index -1 to 36, NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } } }
I20240502 20:45:57.129472 23267 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
I20240502 20:45:57.133127 23133 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 20:45:57.134111 24038 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20240502 20:45:57.138439 24980 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } } }
I20240502 20:45:57.145216 23133 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } } }
I20240502 20:45:57.145818 22978 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index -1 to 26, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:57.149477 23199 consensus_peers.cc:485] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 -> Peer 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917): Couldn't send request to peer 09f2f86617004c96b1e186759506cc68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f6c6fc06ec341e28a4b9f231d78ac30. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.190968 23268 consensus_queue.cc:237] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:45:57.195039 23133 raft_consensus.cc:1261] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240502 20:45:57.197590 24900 consensus_queue.cc:1035] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 35, Time since last communication: 0.000s
W20240502 20:45:57.200080 23199 consensus_peers.cc:485] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 -> Peer b30503059718458bb016185550f5e0fb (127.19.66.133:45729): Couldn't send request to peer b30503059718458bb016185550f5e0fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21b3fcb144894a818b2f649a4473a7ee. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.201320 24038 raft_consensus.cc:2932] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } } }
I20240502 20:45:57.207989 22978 catalog_manager.cc:5484] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index -1 to 37, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:57.213974 23133 raft_consensus.cc:2932] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } } }
I20240502 20:45:57.253243 23131 consensus_queue.cc:237] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:57.256215 23402 raft_consensus.cc:1261] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240502 20:45:57.257107 24134 consensus_queue.cc:1035] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
W20240502 20:45:57.262696 23066 consensus_peers.cc:485] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 -> Peer c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861): Couldn't send request to peer c9af0b94bfe2492bb34c7255879a7328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb9d185d01e44dd49580eeb4ab979346. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.272655 24134 raft_consensus.cc:2932] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.274827 23402 raft_consensus.cc:2932] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.280602 22978 catalog_manager.cc:5484] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: config changed from index -1 to 39, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.355090 23268 consensus_queue.cc:237] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 1.47, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
W20240502 20:45:57.358322 23199 consensus_peers.cc:485] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 -> Peer 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917): Couldn't send request to peer 09f2f86617004c96b1e186759506cc68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df89654f87ed43758ba68a6a5582b059. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.360970 23402 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240502 20:45:57.364347 24910 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 46, Time since last communication: 0.000s
I20240502 20:45:57.370510 24900 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } } }
I20240502 20:45:57.372129 23402 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } } }
I20240502 20:45:57.376295 22978 catalog_manager.cc:5484] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index -1 to 48, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:57.383709 25108 ts_tablet_manager.cc:902] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: Initiating tablet copy from peer ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:57.385808 25108 tablet_copy_client.cc:308] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:39261
I20240502 20:45:57.395987 23424 tablet_copy_service.cc:140] P ee6fe9bcf23a450cb5432a8205660598: Received BeginTabletCopySession request for tablet 711786372ea5495abebec7086f958fef from peer b30503059718458bb016185550f5e0fb ({username='slave'} at 127.19.66.133:36061)
I20240502 20:45:57.396211 23424 tablet_copy_service.cc:161] P ee6fe9bcf23a450cb5432a8205660598: Beginning new tablet copy session on tablet 711786372ea5495abebec7086f958fef from peer b30503059718458bb016185550f5e0fb at {username='slave'} at 127.19.66.133:36061: session id = b30503059718458bb016185550f5e0fb-711786372ea5495abebec7086f958fef
I20240502 20:45:57.398064 23424 tablet_copy_source_session.cc:205] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:57.408118 25108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711786372ea5495abebec7086f958fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.421990 25108 tablet_copy_client.cc:783] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:57.422257 25108 tablet_copy_client.cc:647] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:57.427886 23268 consensus_queue.cc:237] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:45:57.428378 25119 ts_tablet_manager.cc:902] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: Initiating tablet copy from peer ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
W20240502 20:45:57.430235 23199 consensus_peers.cc:485] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 -> Peer b30503059718458bb016185550f5e0fb (127.19.66.133:45729): Couldn't send request to peer b30503059718458bb016185550f5e0fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37ab508cf18c46d8ad1775086354f379. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.438011 23403 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240502 20:45:57.439019 24979 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 20:45:57.443065 25108 tablet_copy_client.cc:523] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:57.445001 24979 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index -1 to 58, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } } }
I20240502 20:45:57.445326 25119 tablet_copy_client.cc:308] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:39261
I20240502 20:45:57.448369 23404 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index -1 to 58, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } } }
I20240502 20:45:57.453251 22978 catalog_manager.cc:5484] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: config changed from index -1 to 58, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } } }
I20240502 20:45:57.470108 25108 tablet_bootstrap.cc:492] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: Bootstrap starting.
I20240502 20:45:57.479631 23130 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 1.44, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
W20240502 20:45:57.482002 23066 consensus_peers.cc:485] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 -> Peer c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861): Couldn't send request to peer c9af0b94bfe2492bb34c7255879a7328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be76bba97022462a86ed4a4cc2a2dd64. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.483637 23424 tablet_copy_service.cc:140] P ee6fe9bcf23a450cb5432a8205660598: Received BeginTabletCopySession request for tablet 5a5fdd7816e345038ed9b6946084a67e from peer c9af0b94bfe2492bb34c7255879a7328 ({username='slave'} at 127.19.66.134:44425)
I20240502 20:45:57.483863 23424 tablet_copy_service.cc:161] P ee6fe9bcf23a450cb5432a8205660598: Beginning new tablet copy session on tablet 5a5fdd7816e345038ed9b6946084a67e from peer c9af0b94bfe2492bb34c7255879a7328 at {username='slave'} at 127.19.66.134:44425: session id = c9af0b94bfe2492bb34c7255879a7328-5a5fdd7816e345038ed9b6946084a67e
I20240502 20:45:57.485368 23424 tablet_copy_source_session.cc:205] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:57.485884 23267 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 44. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240502 20:45:57.486338 24966 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20240502 20:45:57.489405 25119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5fdd7816e345038ed9b6946084a67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.494910 24926 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.495848 23267 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.500185 22980 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: config changed from index -1 to 45, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:57.502117 25108 log.cc:824] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:57.502647 25119 tablet_copy_client.cc:783] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:57.502938 25119 tablet_copy_client.cc:647] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:57.506855 25119 tablet_copy_client.cc:523] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:57.509425 25119 tablet_bootstrap.cc:492] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: Bootstrap starting.
I20240502 20:45:57.512992 25135 ts_tablet_manager.cc:902] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: Initiating tablet copy from peer 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:57.515203 25135 tablet_copy_client.cc:308] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:35415
I20240502 20:45:57.539142 23289 tablet_copy_service.cc:140] P 3db3750c037e4393927784c55f66d349: Received BeginTabletCopySession request for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 from peer 09f2f86617004c96b1e186759506cc68 ({username='slave'} at 127.19.66.135:38529)
I20240502 20:45:57.539350 23289 tablet_copy_service.cc:161] P 3db3750c037e4393927784c55f66d349: Beginning new tablet copy session on tablet 0f6c6fc06ec341e28a4b9f231d78ac30 from peer 09f2f86617004c96b1e186759506cc68 at {username='slave'} at 127.19.66.135:38529: session id = 09f2f86617004c96b1e186759506cc68-0f6c6fc06ec341e28a4b9f231d78ac30
I20240502 20:45:57.543412 23289 tablet_copy_source_session.cc:205] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:57.543897 25119 log.cc:824] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:57.546319 25135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6c6fc06ec341e28a4b9f231d78ac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.549644 23402 consensus_queue.cc:237] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
I20240502 20:45:57.551960 23130 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240502 20:45:57.552574 24994 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240502 20:45:57.554327 25135 tablet_copy_client.cc:783] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:57.554587 25135 tablet_copy_client.cc:647] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:57.558655 25135 tablet_copy_client.cc:523] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:57.560729 25135 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: Bootstrap starting.
W20240502 20:45:57.578886 23333 consensus_peers.cc:485] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 -> Peer 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917): Couldn't send request to peer 09f2f86617004c96b1e186759506cc68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b80c0d7e5de4ad58511694ec615dfef. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.584089 24994 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } } }
I20240502 20:45:57.589533 23132 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } } }
I20240502 20:45:57.590291 22978 catalog_manager.cc:5484] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: config changed from index -1 to 69, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:57.598223 25146 raft_consensus.cc:482] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:45:57.598502 25146 raft_consensus.cc:504] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:57.604038 25146 leader_election.cc:288] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132:45001)
I20240502 20:45:57.620249 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d54cbc40bcc4fa481a04d100ed3c927" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" is_pre_election: true
I20240502 20:45:57.620609 24422 raft_consensus.cc:2454] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 1.
I20240502 20:45:57.621294 24757 leader_election.cc:302] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09f2f86617004c96b1e186759506cc68, 94ac80c7dd7f4767bb357b89c547d8b2; no voters: 
I20240502 20:45:57.621600 25146 raft_consensus.cc:2781] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 20:45:57.621716 25146 raft_consensus.cc:482] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:45:57.621764 25146 raft_consensus.cc:3037] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:57.623497 25146 raft_consensus.cc:504] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:57.623948 25146 leader_election.cc:288] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 2 election: Requested vote from peers 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132:45001)
I20240502 20:45:57.625049 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d54cbc40bcc4fa481a04d100ed3c927" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "94ac80c7dd7f4767bb357b89c547d8b2"
I20240502 20:45:57.625269 24422 raft_consensus.cc:3037] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:57.628085 24422 raft_consensus.cc:2454] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 2.
I20240502 20:45:57.628844 24757 leader_election.cc:302] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09f2f86617004c96b1e186759506cc68, 94ac80c7dd7f4767bb357b89c547d8b2; no voters: 
I20240502 20:45:57.629258 25146 raft_consensus.cc:2781] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:57.629557 25146 raft_consensus.cc:686] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Becoming Leader. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Running, Role: LEADER
I20240502 20:45:57.629961 25146 consensus_queue.cc:237] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } }
I20240502 20:45:57.633726 22980 catalog_manager.cc:5484] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:57.664857 25135 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=716 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.665694 25135 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: Bootstrap complete.
I20240502 20:45:57.666136 25135 ts_tablet_manager.cc:1367] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: Time spent bootstrapping tablet: real 0.105s	user 0.062s	sys 0.004s
I20240502 20:45:57.667043 25135 raft_consensus.cc:348] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
I20240502 20:45:57.667340 25135 raft_consensus.cc:729] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Initialized, Role: LEARNER
I20240502 20:45:57.667524 25135 consensus_queue.cc:260] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
I20240502 20:45:57.668025 25135 ts_tablet_manager.cc:1398] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:57.670218 23289 tablet_copy_service.cc:342] P 3db3750c037e4393927784c55f66d349: Request end of tablet copy session 09f2f86617004c96b1e186759506cc68-0f6c6fc06ec341e28a4b9f231d78ac30 received from {username='slave'} at 127.19.66.135:38529
I20240502 20:45:57.670373 23289 tablet_copy_service.cc:433] P 3db3750c037e4393927784c55f66d349: ending tablet copy session 09f2f86617004c96b1e186759506cc68-0f6c6fc06ec341e28a4b9f231d78ac30 on tablet 0f6c6fc06ec341e28a4b9f231d78ac30 with peer 09f2f86617004c96b1e186759506cc68
I20240502 20:45:57.674146 25155 ts_tablet_manager.cc:902] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328: Initiating tablet copy from peer 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:57.674926 25108 tablet_bootstrap.cc:492] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=44 ignored=0} inserts{seen=675 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:45:57.674973 25155 tablet_copy_client.cc:308] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:45439
I20240502 20:45:57.675508 25108 tablet_bootstrap.cc:492] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: Bootstrap complete.
I20240502 20:45:57.678716 25160 ts_tablet_manager.cc:902] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2: Initiating tablet copy from peer 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:57.679791 25160 tablet_copy_client.cc:308] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:45439
I20240502 20:45:57.682776 25108 ts_tablet_manager.cc:1367] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: Time spent bootstrapping tablet: real 0.215s	user 0.084s	sys 0.000s
I20240502 20:45:57.691025 23153 tablet_copy_service.cc:140] P 140dd1ea33be47e99530c773c5439d56: Received BeginTabletCopySession request for tablet 98e34e4543d24b46963049828877d3d5 from peer 94ac80c7dd7f4767bb357b89c547d8b2 ({username='slave'} at 127.19.66.132:56675)
I20240502 20:45:57.691229 23153 tablet_copy_service.cc:161] P 140dd1ea33be47e99530c773c5439d56: Beginning new tablet copy session on tablet 98e34e4543d24b46963049828877d3d5 from peer 94ac80c7dd7f4767bb357b89c547d8b2 at {username='slave'} at 127.19.66.132:56675: session id = 94ac80c7dd7f4767bb357b89c547d8b2-98e34e4543d24b46963049828877d3d5
I20240502 20:45:57.692461 23154 tablet_copy_service.cc:140] P 140dd1ea33be47e99530c773c5439d56: Received BeginTabletCopySession request for tablet fb9d185d01e44dd49580eeb4ab979346 from peer c9af0b94bfe2492bb34c7255879a7328 ({username='slave'} at 127.19.66.134:50153)
I20240502 20:45:57.692595 23154 tablet_copy_service.cc:161] P 140dd1ea33be47e99530c773c5439d56: Beginning new tablet copy session on tablet fb9d185d01e44dd49580eeb4ab979346 from peer c9af0b94bfe2492bb34c7255879a7328 at {username='slave'} at 127.19.66.134:50153: session id = c9af0b94bfe2492bb34c7255879a7328-fb9d185d01e44dd49580eeb4ab979346
I20240502 20:45:57.692755 23153 tablet_copy_source_session.cc:205] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:57.693625 23154 tablet_copy_source_session.cc:205] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:57.695587 25160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e34e4543d24b46963049828877d3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.697695 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d185d01e44dd49580eeb4ab979346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.700286 25108 raft_consensus.cc:348] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:45:57.700879 25155 tablet_copy_client.cc:783] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:57.701510 25155 tablet_copy_client.cc:647] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:57.702215 25108 raft_consensus.cc:729] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b30503059718458bb016185550f5e0fb, State: Initialized, Role: LEARNER
I20240502 20:45:57.702715 25108 consensus_queue.cc:260] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 1.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:45:57.703876 25108 ts_tablet_manager.cc:1398] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20240502 20:45:57.705896 25160 tablet_copy_client.cc:783] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:57.706192 25160 tablet_copy_client.cc:647] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:57.707715 25119 tablet_bootstrap.cc:492] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=716 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.707789 23424 tablet_copy_service.cc:342] P ee6fe9bcf23a450cb5432a8205660598: Request end of tablet copy session b30503059718458bb016185550f5e0fb-711786372ea5495abebec7086f958fef received from {username='slave'} at 127.19.66.133:36061
I20240502 20:45:57.707890 23424 tablet_copy_service.cc:433] P ee6fe9bcf23a450cb5432a8205660598: ending tablet copy session b30503059718458bb016185550f5e0fb-711786372ea5495abebec7086f958fef on tablet 711786372ea5495abebec7086f958fef with peer b30503059718458bb016185550f5e0fb
I20240502 20:45:57.708194 25119 tablet_bootstrap.cc:492] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: Bootstrap complete.
I20240502 20:45:57.708747 25119 ts_tablet_manager.cc:1367] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: Time spent bootstrapping tablet: real 0.200s	user 0.076s	sys 0.005s
I20240502 20:45:57.709659 25108 ts_tablet_manager.cc:902] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb: Initiating tablet copy from peer 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:57.710148 25108 tablet_copy_client.cc:308] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:35415
I20240502 20:45:57.712852 25119 raft_consensus.cc:348] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:57.713197 25119 raft_consensus.cc:729] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9af0b94bfe2492bb34c7255879a7328, State: Initialized, Role: LEARNER
I20240502 20:45:57.713433 25160 tablet_copy_client.cc:523] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:57.713630 25119 consensus_queue.cc:260] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 1.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:57.714731 25119 ts_tablet_manager.cc:1398] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 20:45:57.715377 25160 tablet_bootstrap.cc:492] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap starting.
I20240502 20:45:57.717216 23424 tablet_copy_service.cc:342] P ee6fe9bcf23a450cb5432a8205660598: Request end of tablet copy session c9af0b94bfe2492bb34c7255879a7328-5a5fdd7816e345038ed9b6946084a67e received from {username='slave'} at 127.19.66.134:44425
I20240502 20:45:57.717327 23424 tablet_copy_service.cc:433] P ee6fe9bcf23a450cb5432a8205660598: ending tablet copy session c9af0b94bfe2492bb34c7255879a7328-5a5fdd7816e345038ed9b6946084a67e on tablet 5a5fdd7816e345038ed9b6946084a67e with peer c9af0b94bfe2492bb34c7255879a7328
I20240502 20:45:57.722680 23289 tablet_copy_service.cc:140] P 3db3750c037e4393927784c55f66d349: Received BeginTabletCopySession request for tablet 21b3fcb144894a818b2f649a4473a7ee from peer b30503059718458bb016185550f5e0fb ({username='slave'} at 127.19.66.133:55257)
I20240502 20:45:57.722846 23289 tablet_copy_service.cc:161] P 3db3750c037e4393927784c55f66d349: Beginning new tablet copy session on tablet 21b3fcb144894a818b2f649a4473a7ee from peer b30503059718458bb016185550f5e0fb at {username='slave'} at 127.19.66.133:55257: session id = b30503059718458bb016185550f5e0fb-21b3fcb144894a818b2f649a4473a7ee
I20240502 20:45:57.723987 23289 tablet_copy_source_session.cc:205] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:57.727687 25155 tablet_copy_client.cc:523] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:57.728380 23130 consensus_queue.cc:237] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:57.729671 25155 tablet_bootstrap.cc:492] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328: Bootstrap starting.
I20240502 20:45:57.730530 25108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b3fcb144894a818b2f649a4473a7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.731865 23267 raft_consensus.cc:1261] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240502 20:45:57.733994 24134 consensus_queue.cc:1035] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
W20240502 20:45:57.734527 23066 consensus_peers.cc:485] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 -> Peer c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861): Couldn't send request to peer c9af0b94bfe2492bb34c7255879a7328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 798923743c9d444f917b747309d2285c. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.735042 25108 tablet_copy_client.cc:783] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:57.735263 25108 tablet_copy_client.cc:647] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:57.737816 25108 tablet_copy_client.cc:523] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:57.739446 24966 raft_consensus.cc:2932] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.739949 23267 raft_consensus.cc:2932] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.742239 25108 tablet_bootstrap.cc:492] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb: Bootstrap starting.
I20240502 20:45:57.743502 22978 catalog_manager.cc:5484] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index -1 to 37, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.801365 25155 tablet_bootstrap.cc:492] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=45 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.801841 25155 tablet_bootstrap.cc:492] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328: Bootstrap complete.
I20240502 20:45:57.802114 25155 ts_tablet_manager.cc:1367] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328: Time spent bootstrapping tablet: real 0.073s	user 0.059s	sys 0.005s
I20240502 20:45:57.802830 25155 raft_consensus.cc:348] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:57.803083 25155 raft_consensus.cc:729] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9af0b94bfe2492bb34c7255879a7328, State: Initialized, Role: LEARNER
I20240502 20:45:57.803239 25155 consensus_queue.cc:260] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 1.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:57.805678 25155 ts_tablet_manager.cc:1398] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:45:57.808029 23154 tablet_copy_service.cc:342] P 140dd1ea33be47e99530c773c5439d56: Request end of tablet copy session c9af0b94bfe2492bb34c7255879a7328-fb9d185d01e44dd49580eeb4ab979346 received from {username='slave'} at 127.19.66.134:50153
I20240502 20:45:57.808234 23154 tablet_copy_service.cc:433] P 140dd1ea33be47e99530c773c5439d56: ending tablet copy session c9af0b94bfe2492bb34c7255879a7328-fb9d185d01e44dd49580eeb4ab979346 on tablet fb9d185d01e44dd49580eeb4ab979346 with peer c9af0b94bfe2492bb34c7255879a7328
I20240502 20:45:57.818158 25135 ts_tablet_manager.cc:902] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68: Initiating tablet copy from peer 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:57.818773 25135 tablet_copy_client.cc:308] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:35415
I20240502 20:45:57.819689 23289 tablet_copy_service.cc:140] P 3db3750c037e4393927784c55f66d349: Received BeginTabletCopySession request for tablet df89654f87ed43758ba68a6a5582b059 from peer 09f2f86617004c96b1e186759506cc68 ({username='slave'} at 127.19.66.135:38529)
I20240502 20:45:57.819850 23289 tablet_copy_service.cc:161] P 3db3750c037e4393927784c55f66d349: Beginning new tablet copy session on tablet df89654f87ed43758ba68a6a5582b059 from peer 09f2f86617004c96b1e186759506cc68 at {username='slave'} at 127.19.66.135:38529: session id = 09f2f86617004c96b1e186759506cc68-df89654f87ed43758ba68a6a5582b059
I20240502 20:45:57.820737 25160 tablet_bootstrap.cc:492] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=45 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.821143 25160 tablet_bootstrap.cc:492] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap complete.
I20240502 20:45:57.821383 25160 ts_tablet_manager.cc:1367] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2: Time spent bootstrapping tablet: real 0.106s	user 0.069s	sys 0.003s
I20240502 20:45:57.821054 23289 tablet_copy_source_session.cc:205] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:57.822111 25160 raft_consensus.cc:348] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } }
I20240502 20:45:57.822278 25160 raft_consensus.cc:729] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 94ac80c7dd7f4767bb357b89c547d8b2, State: Initialized, Role: LEARNER
I20240502 20:45:57.822412 25160 consensus_queue.cc:260] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 1.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } }
I20240502 20:45:57.822858 25160 ts_tablet_manager.cc:1398] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:57.824065 25135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df89654f87ed43758ba68a6a5582b059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.827875 23154 tablet_copy_service.cc:342] P 140dd1ea33be47e99530c773c5439d56: Request end of tablet copy session 94ac80c7dd7f4767bb357b89c547d8b2-98e34e4543d24b46963049828877d3d5 received from {username='slave'} at 127.19.66.132:56675
I20240502 20:45:57.828004 23154 tablet_copy_service.cc:433] P 140dd1ea33be47e99530c773c5439d56: ending tablet copy session 94ac80c7dd7f4767bb357b89c547d8b2-98e34e4543d24b46963049828877d3d5 on tablet 98e34e4543d24b46963049828877d3d5 with peer 94ac80c7dd7f4767bb357b89c547d8b2
I20240502 20:45:57.831022 25135 tablet_copy_client.cc:783] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:57.831284 25135 tablet_copy_client.cc:647] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:57.833185 25135 tablet_copy_client.cc:523] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:57.837110 25135 tablet_bootstrap.cc:492] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68: Bootstrap starting.
I20240502 20:45:57.850917 23267 consensus_queue.cc:237] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:57.856730 23131 raft_consensus.cc:1261] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240502 20:45:57.858265 24979 consensus_queue.cc:1035] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240502 20:45:57.865586 25095 raft_consensus.cc:2932] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.870122 22978 catalog_manager.cc:5484] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index -1 to 55, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:57.870538 24556 raft_consensus.cc:1203] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.32->[1.33-1.53]   Dedup: 1.45->[1.46-1.53]
I20240502 20:45:57.872520 23131 raft_consensus.cc:2932] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, NON_VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } } }
I20240502 20:45:57.885735 24938 raft_consensus.cc:1050] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598: attempting to promote NON_VOTER b30503059718458bb016185550f5e0fb to VOTER
I20240502 20:45:57.886328 24938 consensus_queue.cc:237] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 1.53, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
I20240502 20:45:57.888031 24556 raft_consensus.cc:1261] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240502 20:45:57.888407 23132 raft_consensus.cc:1261] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240502 20:45:57.888845 24995 consensus_queue.cc:1035] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20240502 20:45:57.889649 25108 tablet_bootstrap.cc:492] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=52 ignored=0} inserts{seen=737 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:45:57.890070 25108 tablet_bootstrap.cc:492] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb: Bootstrap complete.
I20240502 20:45:57.890362 25108 ts_tablet_manager.cc:1367] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb: Time spent bootstrapping tablet: real 0.148s	user 0.072s	sys 0.004s
I20240502 20:45:57.890709 24995 consensus_queue.cc:1035] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20240502 20:45:57.891245 25108 raft_consensus.cc:348] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:45:57.891515 24995 raft_consensus.cc:2932] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Committing config change with OpId 1.54: config changed from index 33 to 54, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:57.891886 25108 raft_consensus.cc:729] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b30503059718458bb016185550f5e0fb, State: Initialized, Role: LEARNER
I20240502 20:45:57.892058 25108 consensus_queue.cc:260] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 52, Last appended: 1.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:45:57.892453 25108 ts_tablet_manager.cc:1398] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 20:45:57.893239 23289 tablet_copy_service.cc:342] P 3db3750c037e4393927784c55f66d349: Request end of tablet copy session b30503059718458bb016185550f5e0fb-21b3fcb144894a818b2f649a4473a7ee received from {username='slave'} at 127.19.66.133:55257
I20240502 20:45:57.893352 23289 tablet_copy_service.cc:433] P 3db3750c037e4393927784c55f66d349: ending tablet copy session b30503059718458bb016185550f5e0fb-21b3fcb144894a818b2f649a4473a7ee on tablet 21b3fcb144894a818b2f649a4473a7ee with peer b30503059718458bb016185550f5e0fb
I20240502 20:45:57.896453 24556 raft_consensus.cc:2932] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index 33 to 54, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:57.899839 24690 raft_consensus.cc:1203] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.33->[1.34-1.50]   Dedup: 1.44->[1.45-1.50]
I20240502 20:45:57.900687 23133 raft_consensus.cc:2932] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index 33 to 54, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:57.901433 22980 catalog_manager.cc:5484] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: config changed from index 33 to 54, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:57.916173 23400 consensus_queue.cc:237] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
I20240502 20:45:57.918184 24556 raft_consensus.cc:1261] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240502 20:45:57.918738 24938 consensus_queue.cc:1035] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240502 20:45:57.921015 24938 raft_consensus.cc:2932] T 711786372ea5495abebec7086f958fef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index 54 to 55, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:57.923919 24556 raft_consensus.cc:2932] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index 54 to 55, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:57.926048 25155 ts_tablet_manager.cc:902] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: Initiating tablet copy from peer 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:57.926707 25155 tablet_copy_client.cc:308] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:45439
I20240502 20:45:57.927301 23154 tablet_copy_service.cc:140] P 140dd1ea33be47e99530c773c5439d56: Received BeginTabletCopySession request for tablet be76bba97022462a86ed4a4cc2a2dd64 from peer c9af0b94bfe2492bb34c7255879a7328 ({username='slave'} at 127.19.66.134:50153)
I20240502 20:45:57.927408 23154 tablet_copy_service.cc:161] P 140dd1ea33be47e99530c773c5439d56: Beginning new tablet copy session on tablet be76bba97022462a86ed4a4cc2a2dd64 from peer c9af0b94bfe2492bb34c7255879a7328 at {username='slave'} at 127.19.66.134:50153: session id = c9af0b94bfe2492bb34c7255879a7328-be76bba97022462a86ed4a4cc2a2dd64
I20240502 20:45:57.927467 22985 catalog_manager.cc:5484] T 711786372ea5495abebec7086f958fef P b30503059718458bb016185550f5e0fb reported cstate change: config changed from index 54 to 55, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:57.928802 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 711786372ea5495abebec7086f958fef with cas_config_opid_index 54: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:57.930272 23113 tablet_service.cc:1514] Processing DeleteTablet for tablet 711786372ea5495abebec7086f958fef with delete_type TABLET_DATA_TOMBSTONED (TS 140dd1ea33be47e99530c773c5439d56 not found in new config with opid_index 55) from {username='slave'} at 127.0.0.1:33346
I20240502 20:45:57.930606 23154 tablet_copy_source_session.cc:205] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:57.931887 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be76bba97022462a86ed4a4cc2a2dd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:57.934971 25155 tablet_copy_client.cc:783] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:57.935171 25155 tablet_copy_client.cc:647] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:57.937530 25155 tablet_copy_client.cc:523] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:57.939466 25155 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: Bootstrap starting.
I20240502 20:45:57.949116 25068 raft_consensus.cc:1050] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: attempting to promote NON_VOTER c9af0b94bfe2492bb34c7255879a7328 to VOTER
I20240502 20:45:57.949595 25068 consensus_queue.cc:237] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.51, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:57.950981 23267 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch)
I20240502 20:45:57.951725 24032 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 20:45:57.951923 24690 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch)
I20240502 20:45:57.954402 24032 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Committing config change with OpId 1.52: config changed from index 34 to 52, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:57.956751 23267 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index 34 to 52, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:57.960623 22985 catalog_manager.cc:5484] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index 34 to 52, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:57.964650 25135 tablet_bootstrap.cc:492] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68: Bootstrap replayed 1/1 log segments. Stats: ops{read=59 overwritten=0 applied=59 ignored=0} inserts{seen=731 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:57.965026 25135 tablet_bootstrap.cc:492] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68: Bootstrap complete.
I20240502 20:45:57.965271 25135 ts_tablet_manager.cc:1367] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68: Time spent bootstrapping tablet: real 0.128s	user 0.070s	sys 0.008s
I20240502 20:45:57.965780 25068 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 20:45:57.965832 25135 raft_consensus.cc:348] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
I20240502 20:45:57.965932 25135 raft_consensus.cc:729] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Initialized, Role: LEARNER
I20240502 20:45:57.966038 25135 consensus_queue.cc:260] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 59, Last appended: 1.59, Last appended by leader: 59, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
I20240502 20:45:57.966415 25135 ts_tablet_manager.cc:1398] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:57.967145 23289 tablet_copy_service.cc:342] P 3db3750c037e4393927784c55f66d349: Request end of tablet copy session 09f2f86617004c96b1e186759506cc68-df89654f87ed43758ba68a6a5582b059 received from {username='slave'} at 127.19.66.135:38529
I20240502 20:45:57.967239 23289 tablet_copy_service.cc:433] P 3db3750c037e4393927784c55f66d349: ending tablet copy session 09f2f86617004c96b1e186759506cc68-df89654f87ed43758ba68a6a5582b059 on tablet df89654f87ed43758ba68a6a5582b059 with peer 09f2f86617004c96b1e186759506cc68
I20240502 20:45:57.967414 24690 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index 34 to 52, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:57.976410 25212 tablet_replica.cc:330] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56: stopping tablet replica
I20240502 20:45:57.976825 25212 raft_consensus.cc:2229] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:57.977289 25212 raft_consensus.cc:2258] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 20:45:57.985534 23202 consensus_peers.cc:485] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 -> Peer c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861): Couldn't send request to peer c9af0b94bfe2492bb34c7255879a7328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96ee320aaf2c40a9885f934c5c653c78. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:45:57.997972 25212 ts_tablet_manager.cc:1875] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:58.001880 25212 ts_tablet_manager.cc:1888] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.54
I20240502 20:45:58.002035 25212 log.cc:1196] T 711786372ea5495abebec7086f958fef P 140dd1ea33be47e99530c773c5439d56: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/711786372ea5495abebec7086f958fef
I20240502 20:45:58.002974 22972 catalog_manager.cc:4830] TS 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439): tablet 711786372ea5495abebec7086f958fef (table rebalance_test_table_2 [id=16f6b8a52dc94e42a53292a33382ae40]) successfully deleted
I20240502 20:45:58.005054 25108 ts_tablet_manager.cc:902] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb: Initiating tablet copy from peer 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:58.005638 25108 tablet_copy_client.cc:308] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:35415
I20240502 20:45:58.006546 23289 tablet_copy_service.cc:140] P 3db3750c037e4393927784c55f66d349: Received BeginTabletCopySession request for tablet 37ab508cf18c46d8ad1775086354f379 from peer b30503059718458bb016185550f5e0fb ({username='slave'} at 127.19.66.133:55257)
I20240502 20:45:58.006695 23289 tablet_copy_service.cc:161] P 3db3750c037e4393927784c55f66d349: Beginning new tablet copy session on tablet 37ab508cf18c46d8ad1775086354f379 from peer b30503059718458bb016185550f5e0fb at {username='slave'} at 127.19.66.133:55257: session id = b30503059718458bb016185550f5e0fb-37ab508cf18c46d8ad1775086354f379
I20240502 20:45:58.007776 23289 tablet_copy_source_session.cc:205] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:58.010604 25108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab508cf18c46d8ad1775086354f379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.013754 25108 tablet_copy_client.cc:783] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:58.013988 25108 tablet_copy_client.cc:647] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:58.017282 24556 raft_consensus.cc:1203] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.36->[1.37-1.56]   Dedup: 1.53->[1.54-1.56]
I20240502 20:45:58.023566 25135 ts_tablet_manager.cc:902] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: Initiating tablet copy from peer ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:58.024153 25135 tablet_copy_client.cc:308] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:39261
I20240502 20:45:58.037278 23404 consensus_queue.cc:237] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 1.53, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:58.038094 23424 tablet_copy_service.cc:140] P ee6fe9bcf23a450cb5432a8205660598: Received BeginTabletCopySession request for tablet 4b80c0d7e5de4ad58511694ec615dfef from peer 09f2f86617004c96b1e186759506cc68 ({username='slave'} at 127.19.66.135:43975)
I20240502 20:45:58.038219 23424 tablet_copy_service.cc:161] P ee6fe9bcf23a450cb5432a8205660598: Beginning new tablet copy session on tablet 4b80c0d7e5de4ad58511694ec615dfef from peer 09f2f86617004c96b1e186759506cc68 at {username='slave'} at 127.19.66.135:43975: session id = 09f2f86617004c96b1e186759506cc68-4b80c0d7e5de4ad58511694ec615dfef
I20240502 20:45:58.038448 25108 tablet_copy_client.cc:523] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:58.039342 23424 tablet_copy_source_session.cc:205] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:58.040277 25108 tablet_bootstrap.cc:492] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb: Bootstrap starting.
I20240502 20:45:58.040835 24690 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240502 20:45:58.042790 24995 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20240502 20:45:58.042999 25135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b80c0d7e5de4ad58511694ec615dfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.049006 25068 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Committing config change with OpId 1.54: config changed from index 52 to 54, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.049396 25095 raft_consensus.cc:1050] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349: attempting to promote NON_VOTER b30503059718458bb016185550f5e0fb to VOTER
I20240502 20:45:58.049890 25095 consensus_queue.cc:237] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
I20240502 20:45:58.051031 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a5fdd7816e345038ed9b6946084a67e with cas_config_opid_index 52: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:58.052614 24556 raft_consensus.cc:1261] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240502 20:45:58.053041 24910 consensus_queue.cc:1035] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 20:45:58.053278 22985 catalog_manager.cc:5484] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: config changed from index 52 to 54, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:58.056077 24910 raft_consensus.cc:2932] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index 37 to 58, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:58.057889 25135 tablet_copy_client.cc:783] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:58.058099 25135 tablet_copy_client.cc:647] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:58.059829 24690 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index 52 to 54, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.061429 23133 raft_consensus.cc:1261] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240502 20:45:58.063867 24900 consensus_queue.cc:1035] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 20:45:58.065409 23133 raft_consensus.cc:2932] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index 37 to 58, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:58.068718 24556 raft_consensus.cc:2932] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index 37 to 58, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:58.069917 22978 catalog_manager.cc:5484] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index 37 to 58, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:58.072978 25155 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: Bootstrap replayed 1/1 log segments. Stats: ops{read=48 overwritten=0 applied=47 ignored=0} inserts{seen=732 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:45:58.073462 25155 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: Bootstrap complete.
I20240502 20:45:58.073671 25155 ts_tablet_manager.cc:1367] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: Time spent bootstrapping tablet: real 0.134s	user 0.069s	sys 0.000s
I20240502 20:45:58.074187 25155 raft_consensus.cc:348] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:58.074858 25155 raft_consensus.cc:729] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9af0b94bfe2492bb34c7255879a7328, State: Initialized, Role: LEARNER
I20240502 20:45:58.074983 25155 consensus_queue.cc:260] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 47, Last appended: 1.48, Last appended by leader: 48, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:58.075281 25155 ts_tablet_manager.cc:1398] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:58.077627 23154 tablet_copy_service.cc:342] P 140dd1ea33be47e99530c773c5439d56: Request end of tablet copy session c9af0b94bfe2492bb34c7255879a7328-be76bba97022462a86ed4a4cc2a2dd64 received from {username='slave'} at 127.19.66.134:50153
I20240502 20:45:58.077723 23154 tablet_copy_service.cc:433] P 140dd1ea33be47e99530c773c5439d56: ending tablet copy session c9af0b94bfe2492bb34c7255879a7328-be76bba97022462a86ed4a4cc2a2dd64 on tablet be76bba97022462a86ed4a4cc2a2dd64 with peer c9af0b94bfe2492bb34c7255879a7328
I20240502 20:45:58.081020 23268 consensus_queue.cc:237] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
I20240502 20:45:58.081816 23247 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5fdd7816e345038ed9b6946084a67e with delete_type TABLET_DATA_TOMBSTONED (TS 3db3750c037e4393927784c55f66d349 not found in new config with opid_index 54) from {username='slave'} at 127.0.0.1:44302
I20240502 20:45:58.082324 25239 tablet_replica.cc:330] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349: stopping tablet replica
I20240502 20:45:58.082652 25135 tablet_copy_client.cc:523] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:58.082680 25239 raft_consensus.cc:2229] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:58.082919 25239 raft_consensus.cc:2258] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:58.084594 25135 tablet_bootstrap.cc:492] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: Bootstrap starting.
I20240502 20:45:58.084607 25239 ts_tablet_manager.cc:1875] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:58.085976 24556 raft_consensus.cc:1261] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240502 20:45:58.087509 24038 consensus_queue.cc:1035] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240502 20:45:58.090134 24910 raft_consensus.cc:2932] T 21b3fcb144894a818b2f649a4473a7ee P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index 58 to 59, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:58.090950 24556 raft_consensus.cc:2932] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 58 to 59, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:58.093513 25239 ts_tablet_manager.cc:1888] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.53
I20240502 20:45:58.093667 25239 log.cc:1196] T 5a5fdd7816e345038ed9b6946084a67e P 3db3750c037e4393927784c55f66d349: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/5a5fdd7816e345038ed9b6946084a67e
I20240502 20:45:58.093818 22987 catalog_manager.cc:5484] T 21b3fcb144894a818b2f649a4473a7ee P b30503059718458bb016185550f5e0fb reported cstate change: config changed from index 58 to 59, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:58.095638 22972 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21b3fcb144894a818b2f649a4473a7ee with cas_config_opid_index 58: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:58.095824 22972 catalog_manager.cc:4830] TS 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415): tablet 5a5fdd7816e345038ed9b6946084a67e (table rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]) successfully deleted
I20240502 20:45:58.101153 23113 tablet_service.cc:1514] Processing DeleteTablet for tablet 21b3fcb144894a818b2f649a4473a7ee with delete_type TABLET_DATA_TOMBSTONED (TS 140dd1ea33be47e99530c773c5439d56 not found in new config with opid_index 59) from {username='slave'} at 127.0.0.1:33346
I20240502 20:45:58.101419 25212 tablet_replica.cc:330] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56: stopping tablet replica
I20240502 20:45:58.101552 25212 raft_consensus.cc:2229] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:58.101866 25212 raft_consensus.cc:2258] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:58.102855 25212 ts_tablet_manager.cc:1875] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:58.103353 24422 raft_consensus.cc:1203] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.35->[1.36-1.47]   Dedup: 1.45->[1.46-1.47]
I20240502 20:45:58.105057 25212 ts_tablet_manager.cc:1888] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.58
I20240502 20:45:58.105181 25212 log.cc:1196] T 21b3fcb144894a818b2f649a4473a7ee P 140dd1ea33be47e99530c773c5439d56: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/21b3fcb144894a818b2f649a4473a7ee
I20240502 20:45:58.115103 22972 catalog_manager.cc:4830] TS 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439): tablet 21b3fcb144894a818b2f649a4473a7ee (table rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]) successfully deleted
I20240502 20:45:58.117753 24824 raft_consensus.cc:1203] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.37]   Dedup: 1.31->[1.32-1.37]
I20240502 20:45:58.150460 24926 raft_consensus.cc:1050] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56: attempting to promote NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 to VOTER
I20240502 20:45:58.150986 24926 consensus_queue.cc:237] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } }
I20240502 20:45:58.152576 24422 raft_consensus.cc:1261] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 LEARNER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240502 20:45:58.153337 24134 consensus_queue.cc:1035] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 47, Time since last communication: 0.000s
I20240502 20:45:58.153390 23404 raft_consensus.cc:1261] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240502 20:45:58.155575 24134 consensus_queue.cc:1035] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 47, Time since last communication: 0.000s
I20240502 20:45:58.158799 24134 raft_consensus.cc:2932] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index 36 to 49, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:45:58.162410 23404 raft_consensus.cc:2932] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index 36 to 49, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:45:58.163066 22987 catalog_manager.cc:5484] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: config changed from index 36 to 49, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:58.165614 24422 raft_consensus.cc:2932] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index 36 to 49, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:45:58.175894 24979 raft_consensus.cc:1050] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: attempting to promote NON_VOTER 09f2f86617004c96b1e186759506cc68 to VOTER
I20240502 20:45:58.176327 24979 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:45:58.186177 23130 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240502 20:45:58.188086 24824 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240502 20:45:58.192221 23133 consensus_queue.cc:237] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } }
I20240502 20:45:58.193774 24422 raft_consensus.cc:1261] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240502 20:45:58.193828 24979 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 37, Time since last communication: 0.000s
I20240502 20:45:58.194825 24926 consensus_queue.cc:1035] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 20:45:58.195310 24910 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 37, Time since last communication: 0.000s
I20240502 20:45:58.198343 25240 raft_consensus.cc:2932] T 98e34e4543d24b46963049828877d3d5 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index 49 to 51, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:45:58.200776 22972 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98e34e4543d24b46963049828877d3d5 with cas_config_opid_index 49: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:58.200994 24422 raft_consensus.cc:2932] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index 49 to 51, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:45:58.204222 22980 catalog_manager.cc:5484] T 98e34e4543d24b46963049828877d3d5 P 94ac80c7dd7f4767bb357b89c547d8b2 reported cstate change: config changed from index 49 to 51, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:45:58.204425 24979 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index 26 to 39, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.207713 24824 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 26 to 39, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.208775 23382 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e34e4543d24b46963049828877d3d5 with delete_type TABLET_DATA_TOMBSTONED (TS ee6fe9bcf23a450cb5432a8205660598 not found in new config with opid_index 51) from {username='slave'} at 127.0.0.1:39856
I20240502 20:45:58.209878 22985 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index 26 to 39, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:45:58.210093 23130 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 26 to 39, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.222663 25257 tablet_replica.cc:330] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598: stopping tablet replica
I20240502 20:45:58.223340 25257 raft_consensus.cc:2229] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:58.223573 25257 raft_consensus.cc:2258] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:58.225047 25257 ts_tablet_manager.cc:1875] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:58.233809 25257 ts_tablet_manager.cc:1888] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.50
I20240502 20:45:58.237653 24422 raft_consensus.cc:1261] T 4d54cbc40bcc4fa481a04d100ed3c927 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 20:45:58.237860 25257 log.cc:1196] T 98e34e4543d24b46963049828877d3d5 P ee6fe9bcf23a450cb5432a8205660598: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/98e34e4543d24b46963049828877d3d5
I20240502 20:45:58.238337 25153 consensus_queue.cc:1035] T 4d54cbc40bcc4fa481a04d100ed3c927 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:45:58.239712 22971 catalog_manager.cc:4830] TS ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261): tablet 98e34e4543d24b46963049828877d3d5 (table rebalance_test_table_3 [id=e6d084e24e8342d892514572e869edd1]) successfully deleted
I20240502 20:45:58.249848 25108 tablet_bootstrap.cc:492] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=64 ignored=0} inserts{seen=736 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:58.250200 25108 tablet_bootstrap.cc:492] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb: Bootstrap complete.
I20240502 20:45:58.250435 25108 ts_tablet_manager.cc:1367] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb: Time spent bootstrapping tablet: real 0.210s	user 0.075s	sys 0.007s
I20240502 20:45:58.250960 25108 raft_consensus.cc:348] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:45:58.251097 25108 raft_consensus.cc:729] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b30503059718458bb016185550f5e0fb, State: Initialized, Role: LEARNER
I20240502 20:45:58.251201 25108 consensus_queue.cc:260] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:45:58.266124 25108 ts_tablet_manager.cc:1398] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb: Time spent starting tablet: real 0.016s	user 0.002s	sys 0.000s
I20240502 20:45:58.266927 23289 tablet_copy_service.cc:342] P 3db3750c037e4393927784c55f66d349: Request end of tablet copy session b30503059718458bb016185550f5e0fb-37ab508cf18c46d8ad1775086354f379 received from {username='slave'} at 127.19.66.133:55257
I20240502 20:45:58.267040 23289 tablet_copy_service.cc:433] P 3db3750c037e4393927784c55f66d349: ending tablet copy session b30503059718458bb016185550f5e0fb-37ab508cf18c46d8ad1775086354f379 on tablet 37ab508cf18c46d8ad1775086354f379 with peer b30503059718458bb016185550f5e0fb
I20240502 20:45:58.270512 24690 raft_consensus.cc:1203] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.38->[1.39-1.52]   Dedup: 1.45->[1.46-1.52]
I20240502 20:45:58.296422 24967 raft_consensus.cc:1050] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56: attempting to promote NON_VOTER c9af0b94bfe2492bb34c7255879a7328 to VOTER
I20240502 20:45:58.296911 24967 consensus_queue.cc:237] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.53, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:58.298602 24690 raft_consensus.cc:1261] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240502 20:45:58.299043 23398 raft_consensus.cc:1261] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240502 20:45:58.299058 24966 consensus_queue.cc:1035] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 52, Time since last communication: 0.000s
I20240502 20:45:58.301178 24966 consensus_queue.cc:1035] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 52, Time since last communication: 0.000s
I20240502 20:45:58.302954 24926 raft_consensus.cc:2932] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.54: config changed from index 39 to 54, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.304059 24824 raft_consensus.cc:1203] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.47->[1.48-1.68]   Dedup: 1.59->[1.60-1.68]
I20240502 20:45:58.304819 23398 raft_consensus.cc:2932] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index 39 to 54, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.305938 24690 raft_consensus.cc:2932] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index 39 to 54, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.306989 25155 ts_tablet_manager.cc:902] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328: Initiating tablet copy from peer 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439)
I20240502 20:45:58.306946 22987 catalog_manager.cc:5484] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: config changed from index 39 to 54, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:58.307545 25155 tablet_copy_client.cc:308] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:45439
I20240502 20:45:58.308643 23154 tablet_copy_service.cc:140] P 140dd1ea33be47e99530c773c5439d56: Received BeginTabletCopySession request for tablet 798923743c9d444f917b747309d2285c from peer c9af0b94bfe2492bb34c7255879a7328 ({username='slave'} at 127.19.66.134:50153)
I20240502 20:45:58.308761 23154 tablet_copy_service.cc:161] P 140dd1ea33be47e99530c773c5439d56: Beginning new tablet copy session on tablet 798923743c9d444f917b747309d2285c from peer c9af0b94bfe2492bb34c7255879a7328 at {username='slave'} at 127.19.66.134:50153: session id = c9af0b94bfe2492bb34c7255879a7328-798923743c9d444f917b747309d2285c
I20240502 20:45:58.309860 23154 tablet_copy_source_session.cc:205] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:58.312929 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798923743c9d444f917b747309d2285c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.315083 25155 tablet_copy_client.cc:783] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:58.315286 25155 tablet_copy_client.cc:647] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:58.323195 23133 consensus_queue.cc:237] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:58.328217 24690 raft_consensus.cc:1261] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240502 20:45:58.328609 25155 tablet_copy_client.cc:523] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:58.328796 24966 consensus_queue.cc:1035] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240502 20:45:58.333438 24966 raft_consensus.cc:2932] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index 54 to 55, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.336663 25135 tablet_bootstrap.cc:492] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=81 ignored=0} inserts{seen=712 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:45:58.336881 22972 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fb9d185d01e44dd49580eeb4ab979346 with cas_config_opid_index 54: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:58.337007 25135 tablet_bootstrap.cc:492] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: Bootstrap complete.
I20240502 20:45:58.337239 25135 ts_tablet_manager.cc:1367] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: Time spent bootstrapping tablet: real 0.253s	user 0.074s	sys 0.018s
I20240502 20:45:58.337350 24690 raft_consensus.cc:2932] T fb9d185d01e44dd49580eeb4ab979346 P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index 54 to 55, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.337852 25135 raft_consensus.cc:348] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
I20240502 20:45:58.337800 22980 catalog_manager.cc:5484] T fb9d185d01e44dd49580eeb4ab979346 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: config changed from index 54 to 55, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:58.337951 25135 raft_consensus.cc:729] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Initialized, Role: LEARNER
I20240502 20:45:58.338066 25135 consensus_queue.cc:260] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 81, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
I20240502 20:45:58.338395 25135 ts_tablet_manager.cc:1398] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:58.338768 25284 ts_tablet_manager.cc:902] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328: Initiating tablet copy from peer 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:45:58.339061 23424 tablet_copy_service.cc:342] P ee6fe9bcf23a450cb5432a8205660598: Request end of tablet copy session 09f2f86617004c96b1e186759506cc68-4b80c0d7e5de4ad58511694ec615dfef received from {username='slave'} at 127.19.66.135:43975
I20240502 20:45:58.339159 23424 tablet_copy_service.cc:433] P ee6fe9bcf23a450cb5432a8205660598: ending tablet copy session 09f2f86617004c96b1e186759506cc68-4b80c0d7e5de4ad58511694ec615dfef on tablet 4b80c0d7e5de4ad58511694ec615dfef with peer 09f2f86617004c96b1e186759506cc68
I20240502 20:45:58.339396 25284 tablet_copy_client.cc:308] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:35415
I20240502 20:45:58.341483 23382 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9d185d01e44dd49580eeb4ab979346 with delete_type TABLET_DATA_TOMBSTONED (TS ee6fe9bcf23a450cb5432a8205660598 not found in new config with opid_index 55) from {username='slave'} at 127.0.0.1:39856
I20240502 20:45:58.341750 25257 tablet_replica.cc:330] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598: stopping tablet replica
I20240502 20:45:58.341904 25257 raft_consensus.cc:2229] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:58.342136 25257 raft_consensus.cc:2258] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:58.343519 25257 ts_tablet_manager.cc:1875] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:58.346328 25257 ts_tablet_manager.cc:1888] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.54
I20240502 20:45:58.346426 25257 log.cc:1196] T fb9d185d01e44dd49580eeb4ab979346 P ee6fe9bcf23a450cb5432a8205660598: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/fb9d185d01e44dd49580eeb4ab979346
I20240502 20:45:58.347124 22971 catalog_manager.cc:4830] TS ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261): tablet fb9d185d01e44dd49580eeb4ab979346 (table rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]) successfully deleted
I20240502 20:45:58.347265 25155 tablet_bootstrap.cc:492] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328: Bootstrap starting.
I20240502 20:45:58.381392 24690 raft_consensus.cc:1203] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.44->[1.45-1.57]   Dedup: 1.48->[1.49-1.57]
I20240502 20:45:58.396050 23289 tablet_copy_service.cc:140] P 3db3750c037e4393927784c55f66d349: Received BeginTabletCopySession request for tablet 96ee320aaf2c40a9885f934c5c653c78 from peer c9af0b94bfe2492bb34c7255879a7328 ({username='slave'} at 127.19.66.134:51875)
I20240502 20:45:58.396239 23289 tablet_copy_service.cc:161] P 3db3750c037e4393927784c55f66d349: Beginning new tablet copy session on tablet 96ee320aaf2c40a9885f934c5c653c78 from peer c9af0b94bfe2492bb34c7255879a7328 at {username='slave'} at 127.19.66.134:51875: session id = c9af0b94bfe2492bb34c7255879a7328-96ee320aaf2c40a9885f934c5c653c78
I20240502 20:45:58.397285 23289 tablet_copy_source_session.cc:205] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:45:58.402375 24555 raft_consensus.cc:1203] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.57->[1.58-1.70]   Dedup: 1.64->[1.65-1.70]
I20240502 20:45:58.410938 25284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ee320aaf2c40a9885f934c5c653c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:58.415539 25284 tablet_copy_client.cc:783] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Starting download of 0 data blocks...
I20240502 20:45:58.415815 25284 tablet_copy_client.cc:647] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Starting download of 1 WAL segments...
I20240502 20:45:58.417760 24900 raft_consensus.cc:1050] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: attempting to promote NON_VOTER 09f2f86617004c96b1e186759506cc68 to VOTER
I20240502 20:45:58.418174 24900 consensus_queue.cc:237] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:45:58.421139 25284 tablet_copy_client.cc:523] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:45:58.422372 24824 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 20:45:58.423121 25284 tablet_bootstrap.cc:492] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328: Bootstrap starting.
I20240502 20:45:58.424448 23404 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 20:45:58.425127 24980 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 20:45:58.425536 24980 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 20:45:58.429484 24038 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 48 to 70, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.432906 23404 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 48 to 70, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.433982 24823 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 48 to 70, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.438889 22980 catalog_manager.cc:5484] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 48 to 70, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.448513 24134 raft_consensus.cc:1050] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: attempting to promote NON_VOTER c9af0b94bfe2492bb34c7255879a7328 to VOTER
I20240502 20:45:58.448972 24134 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:58.450668 24690 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240502 20:45:58.451094 24967 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 20:45:58.453023 24967 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index 45 to 59, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.456228 24690 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 45 to 59, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.456514 22987 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: config changed from index 45 to 59, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:58.462307 23268 consensus_queue.cc:237] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:45:58.469198 24823 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch)
I20240502 20:45:58.469765 25155 tablet_bootstrap.cc:492] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328: Bootstrap replayed 1/1 log segments. Stats: ops{read=47 overwritten=0 applied=46 ignored=0} inserts{seen=731 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:45:58.469766 24979 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240502 20:45:58.470263 25155 tablet_bootstrap.cc:492] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328: Bootstrap complete.
I20240502 20:45:58.470568 25155 ts_tablet_manager.cc:1367] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328: Time spent bootstrapping tablet: real 0.123s	user 0.060s	sys 0.003s
I20240502 20:45:58.471158 25155 raft_consensus.cc:348] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:58.471607 25155 raft_consensus.cc:729] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9af0b94bfe2492bb34c7255879a7328, State: Initialized, Role: LEARNER
I20240502 20:45:58.471812 25155 consensus_queue.cc:260] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 46, Last appended: 1.47, Last appended by leader: 47, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:58.472198 25155 ts_tablet_manager.cc:1398] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:45:58.472062 25095 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index 70 to 73, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.474117 23154 tablet_copy_service.cc:342] P 140dd1ea33be47e99530c773c5439d56: Request end of tablet copy session c9af0b94bfe2492bb34c7255879a7328-798923743c9d444f917b747309d2285c received from {username='slave'} at 127.19.66.134:50153
I20240502 20:45:58.474212 23154 tablet_copy_service.cc:433] P 140dd1ea33be47e99530c773c5439d56: ending tablet copy session c9af0b94bfe2492bb34c7255879a7328-798923743c9d444f917b747309d2285c on tablet 798923743c9d444f917b747309d2285c with peer c9af0b94bfe2492bb34c7255879a7328
I20240502 20:45:58.477663 24824 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 70 to 73, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.482196 22980 catalog_manager.cc:5484] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 70 to 73, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.485754 23268 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240502 20:45:58.486076 23382 tablet_service.cc:1514] Processing DeleteTablet for tablet df89654f87ed43758ba68a6a5582b059 with delete_type TABLET_DATA_TOMBSTONED (TS ee6fe9bcf23a450cb5432a8205660598 not found in new config with opid_index 73) from {username='slave'} at 127.0.0.1:39856
I20240502 20:45:58.486315 25257 tablet_replica.cc:330] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598: stopping tablet replica
I20240502 20:45:58.486487 25257 raft_consensus.cc:2229] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:58.486629 25257 raft_consensus.cc:2258] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:58.487305 25257 ts_tablet_manager.cc:1875] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:58.488796 22972 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df89654f87ed43758ba68a6a5582b059 with cas_config_opid_index 70: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:58.489899 25257 ts_tablet_manager.cc:1888] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.72
I20240502 20:45:58.489986 25257 log.cc:1196] T df89654f87ed43758ba68a6a5582b059 P ee6fe9bcf23a450cb5432a8205660598: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/df89654f87ed43758ba68a6a5582b059
I20240502 20:45:58.490089 25240 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 20:45:58.490829 22971 catalog_manager.cc:4830] TS ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261): tablet df89654f87ed43758ba68a6a5582b059 (table rebalance_test_table_2 [id=16f6b8a52dc94e42a53292a33382ae40]) successfully deleted
I20240502 20:45:58.492765 23267 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 45 to 59, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.519973 23267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30"
dest_uuid: "3db3750c037e4393927784c55f66d349"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:42874
I20240502 20:45:58.520217 23267 raft_consensus.cc:595] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:58.530649 24900 raft_consensus.cc:1050] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: attempting to promote NON_VOTER b30503059718458bb016185550f5e0fb to VOTER
I20240502 20:45:58.531144 24900 consensus_queue.cc:237] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
I20240502 20:45:58.536072 24554 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 1 LEARNER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 20:45:58.543968 24038 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 20:45:58.545759 25284 tablet_bootstrap.cc:492] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=69 ignored=0} inserts{seen=754 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 20:45:58.546151 25284 tablet_bootstrap.cc:492] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328: Bootstrap complete.
I20240502 20:45:58.546461 25284 ts_tablet_manager.cc:1367] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328: Time spent bootstrapping tablet: real 0.123s	user 0.083s	sys 0.007s
I20240502 20:45:58.549587 23404 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 20:45:58.549970 25284 raft_consensus.cc:348] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:58.550289 25284 raft_consensus.cc:729] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9af0b94bfe2492bb34c7255879a7328, State: Initialized, Role: LEARNER
I20240502 20:45:58.550446 25284 consensus_queue.cc:260] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: true } }
I20240502 20:45:58.550762 25284 ts_tablet_manager.cc:1398] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.000s
I20240502 20:45:58.552233 24910 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 58 to 72, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:58.553606 23289 tablet_copy_service.cc:342] P 3db3750c037e4393927784c55f66d349: Request end of tablet copy session c9af0b94bfe2492bb34c7255879a7328-96ee320aaf2c40a9885f934c5c653c78 received from {username='slave'} at 127.19.66.134:51875
I20240502 20:45:58.553721 23289 tablet_copy_service.cc:433] P 3db3750c037e4393927784c55f66d349: ending tablet copy session c9af0b94bfe2492bb34c7255879a7328-96ee320aaf2c40a9885f934c5c653c78 on tablet 96ee320aaf2c40a9885f934c5c653c78 with peer c9af0b94bfe2492bb34c7255879a7328
I20240502 20:45:58.558116 24555 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 58 to 72, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:58.558954 24979 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 20:45:58.561805 22980 catalog_manager.cc:5484] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb reported cstate change: config changed from index 58 to 72, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:58.570394 23400 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 58 to 72, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:58.586611 24823 raft_consensus.cc:1203] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.68->[1.69-1.92]   Dedup: 1.81->[1.82-1.92]
W20240502 20:45:58.616050 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
W20240502 20:45:58.637185 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
W20240502 20:45:58.643056 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
W20240502 20:45:58.650856 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
W20240502 20:45:58.663726 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
W20240502 20:45:58.672087 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.677461 24994 raft_consensus.cc:1050] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: attempting to promote NON_VOTER 09f2f86617004c96b1e186759506cc68 to VOTER
I20240502 20:45:58.678093 24994 consensus_queue.cc:237] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:45:58.682900 24824 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 1 LEARNER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
W20240502 20:45:58.683408 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.684357 24995 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240502 20:45:58.686319 23130 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 20:45:58.687866 24995 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240502 20:45:58.691167 24995 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index 69 to 94, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.693359 24823 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 69 to 94, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
W20240502 20:45:58.696385 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.697759 22980 catalog_manager.cc:5484] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 69 to 94, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.701117 23130 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 69 to 94, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
W20240502 20:45:58.707055 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.711158 24690 raft_consensus.cc:1203] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.54->[1.55-1.75]   Dedup: 1.70->[1.71-1.75]
W20240502 20:45:58.720255 23225 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
W20240502 20:45:58.735107 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.739591 24690 raft_consensus.cc:1203] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.36->[1.37-1.56]   Dedup: 1.47->[1.48-1.56]
I20240502 20:45:58.751024 24910 raft_consensus.cc:1050] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349: attempting to promote NON_VOTER c9af0b94bfe2492bb34c7255879a7328 to VOTER
I20240502 20:45:58.751449 24910 consensus_queue.cc:237] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:58.754806 24689 raft_consensus.cc:1261] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
W20240502 20:45:58.755355 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.755450 24980 consensus_queue.cc:1035] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240502 20:45:58.757405 23133 raft_consensus.cc:1261] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240502 20:45:58.757936 24980 consensus_queue.cc:1035] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240502 20:45:58.759940 24980 raft_consensus.cc:2932] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index 55 to 77, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.761258 23133 raft_consensus.cc:2932] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 55 to 77, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.762033 24689 raft_consensus.cc:2932] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 55 to 77, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.765316 22987 catalog_manager.cc:5484] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: config changed from index 55 to 77, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
W20240502 20:45:58.770269 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.778370 24134 raft_consensus.cc:1050] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56: attempting to promote NON_VOTER c9af0b94bfe2492bb34c7255879a7328 to VOTER
I20240502 20:45:58.778739 24134 consensus_queue.cc:237] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:58.781437 24690 raft_consensus.cc:1261] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328 [term 1 LEARNER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240502 20:45:58.782086 24926 consensus_queue.cc:1035] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 20:45:58.782228 23267 raft_consensus.cc:1261] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240502 20:45:58.783865 24926 consensus_queue.cc:1035] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 20:45:58.784749 24926 raft_consensus.cc:2932] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index 37 to 58, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
W20240502 20:45:58.787550 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.787942 24690 raft_consensus.cc:2932] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index 37 to 58, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.788700 22985 catalog_manager.cc:5484] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 reported cstate change: config changed from index 37 to 58, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:58.792269 23267 raft_consensus.cc:2932] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index 37 to 58, c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
W20240502 20:45:58.807107 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.810299 23130 consensus_queue.cc:237] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:58.818732 24690 raft_consensus.cc:1261] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Refusing update from remote peer 140dd1ea33be47e99530c773c5439d56: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240502 20:45:58.819432 24134 consensus_queue.cc:1035] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 58, Time since last communication: 0.000s
I20240502 20:45:58.822016 24134 raft_consensus.cc:2932] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index 58 to 60, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.824041 22972 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 798923743c9d444f917b747309d2285c with cas_config_opid_index 58: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:58.824616 24690 raft_consensus.cc:2932] T 798923743c9d444f917b747309d2285c P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 58 to 60, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
W20240502 20:45:58.827957 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.830119 22987 catalog_manager.cc:5484] T 798923743c9d444f917b747309d2285c P 140dd1ea33be47e99530c773c5439d56 reported cstate change: config changed from index 58 to 60, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "140dd1ea33be47e99530c773c5439d56" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:58.833241 23268 consensus_queue.cc:237] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:58.834872 23247 tablet_service.cc:1514] Processing DeleteTablet for tablet 798923743c9d444f917b747309d2285c with delete_type TABLET_DATA_TOMBSTONED (TS 3db3750c037e4393927784c55f66d349 not found in new config with opid_index 60) from {username='slave'} at 127.0.0.1:44302
I20240502 20:45:58.844406 24688 raft_consensus.cc:1261] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240502 20:45:58.845474 24980 consensus_queue.cc:1035] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 81, Time since last communication: 0.000s
W20240502 20:45:58.848001 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.849525 24980 raft_consensus.cc:2932] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index 77 to 83, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.852684 24688 raft_consensus.cc:2932] T 96ee320aaf2c40a9885f934c5c653c78 P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 77 to 83, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:58.853600 22972 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96ee320aaf2c40a9885f934c5c653c78 with cas_config_opid_index 77: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:58.854146 22987 catalog_manager.cc:5484] T 96ee320aaf2c40a9885f934c5c653c78 P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index 77 to 83, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:58.860374 23113 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ee320aaf2c40a9885f934c5c653c78 with delete_type TABLET_DATA_TOMBSTONED (TS 140dd1ea33be47e99530c773c5439d56 not found in new config with opid_index 83) from {username='slave'} at 127.0.0.1:33346
I20240502 20:45:58.863021 24910 raft_consensus.cc:979] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: : Instructing follower 09f2f86617004c96b1e186759506cc68 to start an election
I20240502 20:45:58.863176 24910 raft_consensus.cc:1067] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Signalling peer 09f2f86617004c96b1e186759506cc68 to start an election
I20240502 20:45:58.863299 25349 tablet_replica.cc:330] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: stopping tablet replica
I20240502 20:45:58.863492 25349 raft_consensus.cc:2229] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:58.863597 25349 pending_rounds.cc:62] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: Trying to abort 1 pending ops.
I20240502 20:45:58.863895 25349 pending_rounds.cc:68] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: Aborting op as it isn't in flight: id { term: 1 index: 82 } timestamp: 7023340580150009856 op_type: WRITE_OP write_request { tablet_id: "96ee320aaf2c40a9885f934c5c653c78" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\204\210\n\016g=\t\373\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""dff29bdf61aebbb7" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023340580103188480 authz_token { token_data: "\010\257\371\317\261\006\"3\n\005slave\022*\n b45b0775e1d8423585dd5d5a47ec5dbe\020\001\030\001 \001(\001" signature: "<redacted>""\233>`\230\342\231\260\310\360\356\255Sn\346\"\363\267\370\212\013\346\227\013\361\254N\244\277Y\344\312\235e\341\263o\365\r\317$0\010\313\301()\336m\033)`@\010\300`\003\013v\302\212R\273f*" signing_key_seq_num: 0 } } request_id { client_id: "7cb46be82d6c4a7fa19d848d344d21ee" seq_no: 233 first_incomplete_seq_no: 233 attempt_no: 0 }
I20240502 20:45:58.864382 25349 raft_consensus.cc:2258] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:58.865360 25349 ts_tablet_manager.cc:1875] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:58.867149 24824 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30"
dest_uuid: "09f2f86617004c96b1e186759506cc68"
 from {username='slave'} at 127.19.66.130:38299
I20240502 20:45:58.867365 24824 raft_consensus.cc:482] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:58.867458 24824 raft_consensus.cc:3037] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:58.868652 24824 raft_consensus.cc:504] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:45:58.870566 24824 leader_election.cc:288] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 2 election: Requested vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439), 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
W20240502 20:45:58.871093 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Service unavailable: leader transfer in progress
I20240502 20:45:58.872017 23267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: true dest_uuid: "3db3750c037e4393927784c55f66d349"
I20240502 20:45:58.872169 23267 raft_consensus.cc:3032] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:58.872220 23267 raft_consensus.cc:729] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:45:58.872429 23267 consensus_queue.cc:260] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 47, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:45:58.872659 23267 raft_consensus.cc:3037] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:58.872989 25344 tablet_replica.cc:330] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349: stopping tablet replica
I20240502 20:45:58.873144 25344 raft_consensus.cc:2229] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:58.873275 25344 raft_consensus.cc:2258] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:58.873968 23267 raft_consensus.cc:2454] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 2.
I20240502 20:45:58.874356 24755 leader_election.cc:302] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f2f86617004c96b1e186759506cc68, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:45:58.874614 25265 raft_consensus.cc:2781] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:58.874821 25265 raft_consensus.cc:686] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Becoming Leader. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Running, Role: LEADER
I20240502 20:45:58.875020 25265 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 47, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:45:58.879062 22987 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 reported cstate change: term changed from 1 to 2, leader changed from 3db3750c037e4393927784c55f66d349 (127.19.66.130) to 09f2f86617004c96b1e186759506cc68 (127.19.66.135). New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:58.881321 25344 ts_tablet_manager.cc:1875] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:58.883333 25349 ts_tablet_manager.cc:1888] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.82
I20240502 20:45:58.883471 25349 log.cc:1196] T 96ee320aaf2c40a9885f934c5c653c78 P 140dd1ea33be47e99530c773c5439d56: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/96ee320aaf2c40a9885f934c5c653c78
I20240502 20:45:58.884081 23130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: true dest_uuid: "140dd1ea33be47e99530c773c5439d56"
I20240502 20:45:58.884258 23130 raft_consensus.cc:3037] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:58.885527 23130 raft_consensus.cc:2454] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 2.
I20240502 20:45:58.885991 22972 catalog_manager.cc:4830] TS 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439): tablet 96ee320aaf2c40a9885f934c5c653c78 (table rebalance_test_table_0 [id=b45b0775e1d8423585dd5d5a47ec5dbe]) successfully deleted
I20240502 20:45:58.889436 25344 ts_tablet_manager.cc:1888] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.59
I20240502 20:45:58.889694 25344 log.cc:1196] T 798923743c9d444f917b747309d2285c P 3db3750c037e4393927784c55f66d349: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/798923743c9d444f917b747309d2285c
I20240502 20:45:58.891958 22972 catalog_manager.cc:4830] TS 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415): tablet 798923743c9d444f917b747309d2285c (table rebalance_test_table_2 [id=16f6b8a52dc94e42a53292a33382ae40]) successfully deleted
W20240502 20:45:58.895130 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42760: Illegal state: replica 3db3750c037e4393927784c55f66d349 is not leader of this config: current role FOLLOWER
I20240502 20:45:58.910090 23400 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b80c0d7e5de4ad58511694ec615dfef"
dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33110
I20240502 20:45:58.910301 23400 raft_consensus.cc:595] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:45:58.915951 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:58.919260 23091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47724: Illegal state: replica 140dd1ea33be47e99530c773c5439d56 is not leader of this config: current role FOLLOWER
W20240502 20:45:58.922922 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:58.928653 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:58.937218 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
I20240502 20:45:58.943113 23130 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64"
dest_uuid: "140dd1ea33be47e99530c773c5439d56"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47834
I20240502 20:45:58.943377 23130 raft_consensus.cc:595] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:45:58.945670 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:58.955909 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
I20240502 20:45:58.960444 23267 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 2 index: 49. (index mismatch)
I20240502 20:45:58.960561 23130 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 2 index: 49. (index mismatch)
I20240502 20:45:58.961755 25352 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20240502 20:45:58.966253 25352 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
W20240502 20:45:58.968586 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:58.968717 23091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:58.976503 23091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:58.977020 25294 mvcc.cc:205] Tried to move back new op lower bound from 7023340580694925312 to 7023340580354580480. Current Snapshot: MvccSnapshot[applied={T|T < 7023340580694925312 or (T in {7023340580694925312})}]
W20240502 20:45:58.981463 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
I20240502 20:45:58.982786 23267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379"
dest_uuid: "3db3750c037e4393927784c55f66d349"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:42898
I20240502 20:45:58.982941 23267 raft_consensus.cc:595] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:45:58.987287 23091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:58.990010 24823 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 2.51, Last appended by leader: 47, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
W20240502 20:45:58.990929 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:58.993052 23091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:58.994752 24910 raft_consensus.cc:979] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: : Instructing follower ee6fe9bcf23a450cb5432a8205660598 to start an election
I20240502 20:45:58.994901 24979 raft_consensus.cc:1067] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Signalling peer ee6fe9bcf23a450cb5432a8205660598 to start an election
I20240502 20:45:58.995445 23404 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379"
dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
 from {username='slave'} at 127.19.66.130:42673
I20240502 20:45:58.995594 23404 raft_consensus.cc:482] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:58.995652 23404 raft_consensus.cc:3037] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:58.995632 23131 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 51. Preceding OpId from leader: term: 2 index: 52. (index mismatch)
I20240502 20:45:58.997047 25352 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 20:45:58.997177 23404 raft_consensus.cc:504] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
I20240502 20:45:58.997586 23404 leader_election.cc:288] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 2 election: Requested vote from peers 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415), b30503059718458bb016185550f5e0fb (127.19.66.133:45729)
I20240502 20:45:58.998278 24555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379" candidate_uuid: "ee6fe9bcf23a450cb5432a8205660598" candidate_term: 2 candidate_status { last_received { term: 1 index: 86 } } ignore_live_leader: true dest_uuid: "b30503059718458bb016185550f5e0fb"
I20240502 20:45:58.998505 24555 raft_consensus.cc:3037] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:58.999420 25265 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.52: config changed from index 39 to 52, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:58.999928 24555 raft_consensus.cc:2454] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee6fe9bcf23a450cb5432a8205660598 in term 2.
I20240502 20:45:59.000617 23333 leader_election.cc:302] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b30503059718458bb016185550f5e0fb, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:59.000695 23267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379" candidate_uuid: "ee6fe9bcf23a450cb5432a8205660598" candidate_term: 2 candidate_status { last_received { term: 1 index: 86 } } ignore_live_leader: true dest_uuid: "3db3750c037e4393927784c55f66d349"
I20240502 20:45:59.000831 23267 raft_consensus.cc:3032] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:59.000876 23267 raft_consensus.cc:729] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:45:59.001026 25068 raft_consensus.cc:2781] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:59.001137 25068 raft_consensus.cc:686] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Becoming Leader. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Running, Role: LEADER
I20240502 20:45:59.001041 23267 consensus_queue.cc:260] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
I20240502 20:45:59.001279 23267 raft_consensus.cc:3037] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:59.001269 25068 consensus_queue.cc:237] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
I20240502 20:45:59.001252 23130 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Committing config change with OpId 2.52: config changed from index 39 to 52, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:59.002429 23267 raft_consensus.cc:2454] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee6fe9bcf23a450cb5432a8205660598 in term 2.
W20240502 20:45:59.004418 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42756: Illegal state: replica 3db3750c037e4393927784c55f66d349 is not leader of this config: current role FOLLOWER
I20240502 20:45:59.004961 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:59.005427 22987 catalog_manager.cc:5484] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: term changed from 1 to 2, leader changed from 3db3750c037e4393927784c55f66d349 (127.19.66.130) to ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131). New cstate: current_term: 2 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:59.006567 23091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.009102 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
I20240502 20:45:59.006584 22985 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 reported cstate change: config changed from index 39 to 52, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:59.013211 23247 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 with delete_type TABLET_DATA_TOMBSTONED (TS 3db3750c037e4393927784c55f66d349 not found in new config with opid_index 52) from {username='slave'} at 127.0.0.1:44302
I20240502 20:45:59.013707 25344 tablet_replica.cc:330] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: stopping tablet replica
W20240502 20:45:59.016233 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:59.018461 24555 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 2 index: 88. (index mismatch)
I20240502 20:45:59.017302 23267 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 2 index: 88. (index mismatch)
I20240502 20:45:59.018872 25344 raft_consensus.cc:2229] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:59.019047 25344 raft_consensus.cc:2258] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:59.019542 24938 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 20:45:59.019909 24938 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 20:45:59.020082 25344 ts_tablet_manager.cc:1875] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 20:45:59.023253 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
I20240502 20:45:59.027587 23404 consensus_queue.cc:237] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 2.88, Last appended by leader: 86, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
W20240502 20:45:59.029098 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:59.029182 24555 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 2 index: 88. Preceding OpId from leader: term: 2 index: 89. (index mismatch)
I20240502 20:45:59.029733 24032 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240502 20:45:59.033389 24032 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Committing config change with OpId 2.89: config changed from index 72 to 89, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:45:59.034039 25344 ts_tablet_manager.cc:1888] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.50
I20240502 20:45:59.034463 25344 log.cc:1196] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 3db3750c037e4393927784c55f66d349: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/0f6c6fc06ec341e28a4b9f231d78ac30
I20240502 20:45:59.035094 22972 catalog_manager.cc:4830] TS 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415): tablet 0f6c6fc06ec341e28a4b9f231d78ac30 (table rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]) successfully deleted
I20240502 20:45:59.035216 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37ab508cf18c46d8ad1775086354f379 with cas_config_opid_index 72: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:59.038058 22980 catalog_manager.cc:5484] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: config changed from index 72 to 89, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New cstate: current_term: 2 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:59.037266 24555 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Committing config change with OpId 2.89: config changed from index 72 to 89, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
W20240502 20:45:59.039260 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
I20240502 20:45:59.041944 23247 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ab508cf18c46d8ad1775086354f379 with delete_type TABLET_DATA_TOMBSTONED (TS 3db3750c037e4393927784c55f66d349 not found in new config with opid_index 89) from {username='slave'} at 127.0.0.1:44302
W20240502 20:45:59.042353 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:59.042537 25344 tablet_replica.cc:330] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: stopping tablet replica
I20240502 20:45:59.042728 25344 raft_consensus.cc:2229] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:59.042896 25344 raft_consensus.cc:2258] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:59.043800 25344 ts_tablet_manager.cc:1875] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:59.046960 25344 ts_tablet_manager.cc:1888] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.88
I20240502 20:45:59.047194 25344 log.cc:1196] T 37ab508cf18c46d8ad1775086354f379 P 3db3750c037e4393927784c55f66d349: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/37ab508cf18c46d8ad1775086354f379
I20240502 20:45:59.047988 22972 catalog_manager.cc:4830] TS 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415): tablet 37ab508cf18c46d8ad1775086354f379 (table rebalance_test_table_0 [id=b45b0775e1d8423585dd5d5a47ec5dbe]) successfully deleted
W20240502 20:45:59.058058 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.059217 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:59.072013 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.074798 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:59.088689 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.093669 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:59.105255 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.112648 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:59.121986 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.130224 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:59.138424 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.153633 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:59.158061 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:59.168526 23404 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b80c0d7e5de4ad58511694ec615dfef"
dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33120
I20240502 20:45:59.168720 23404 raft_consensus.cc:595] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:45:59.178311 23091 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.179766 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:59.198026 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.203521 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
I20240502 20:45:59.203661 23131 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64"
dest_uuid: "140dd1ea33be47e99530c773c5439d56"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47864
I20240502 20:45:59.203792 23131 raft_consensus.cc:595] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:45:59.218008 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.229622 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:59.240753 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:59.259055 23400 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b80c0d7e5de4ad58511694ec615dfef"
dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33144
I20240502 20:45:59.259209 23400 raft_consensus.cc:595] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Received request to transfer leadership
W20240502 20:45:59.261298 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
W20240502 20:45:59.261590 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.287021 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.288718 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Service unavailable: leader transfer in progress
I20240502 20:45:59.294544 23130 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64"
dest_uuid: "140dd1ea33be47e99530c773c5439d56"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47880
I20240502 20:45:59.294742 23130 raft_consensus.cc:595] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:45:59.296851 24938 raft_consensus.cc:979] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: : Instructing follower 09f2f86617004c96b1e186759506cc68 to start an election
I20240502 20:45:59.296968 24938 raft_consensus.cc:1067] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Signalling peer 09f2f86617004c96b1e186759506cc68 to start an election
I20240502 20:45:59.298151 24823 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b80c0d7e5de4ad58511694ec615dfef"
dest_uuid: "09f2f86617004c96b1e186759506cc68"
 from {username='slave'} at 127.19.66.131:60367
I20240502 20:45:59.298326 24823 raft_consensus.cc:482] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:59.298401 24823 raft_consensus.cc:3037] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:59.305301 24823 raft_consensus.cc:504] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:45:59.306061 24823 leader_election.cc:288] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 2 election: Requested vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439), ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:45:59.306291 23131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b80c0d7e5de4ad58511694ec615dfef" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 2 candidate_status { last_received { term: 1 index: 103 } } ignore_live_leader: true dest_uuid: "140dd1ea33be47e99530c773c5439d56"
I20240502 20:45:59.306429 23131 raft_consensus.cc:3037] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:59.307437 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b80c0d7e5de4ad58511694ec615dfef" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 2 candidate_status { last_received { term: 1 index: 103 } } ignore_live_leader: true dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
I20240502 20:45:59.307587 23400 raft_consensus.cc:3032] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:59.307643 23400 raft_consensus.cc:729] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Running, Role: LEADER
I20240502 20:45:59.307830 23131 raft_consensus.cc:2454] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 2.
I20240502 20:45:59.307842 23400 consensus_queue.cc:260] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 103, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:45:59.308296 23400 raft_consensus.cc:3037] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:59.309525 23400 raft_consensus.cc:2454] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 2.
I20240502 20:45:59.309865 24754 leader_election.cc:302] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f2f86617004c96b1e186759506cc68, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:45:59.310068 25352 raft_consensus.cc:2781] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:59.310186 25352 raft_consensus.cc:686] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Becoming Leader. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Running, Role: LEADER
I20240502 20:45:59.310344 25352 consensus_queue.cc:237] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
W20240502 20:45:59.311545 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:59.313709 22985 catalog_manager.cc:5484] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 reported cstate change: term changed from 1 to 2, leader changed from ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) to 09f2f86617004c96b1e186759506cc68 (127.19.66.135). New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:45:59.318614 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33008: Illegal state: replica ee6fe9bcf23a450cb5432a8205660598 is not leader of this config: current role FOLLOWER
W20240502 20:45:59.340762 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.366210 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47894: Illegal state: replica 140dd1ea33be47e99530c773c5439d56 is not leader of this config: current role FOLLOWER
W20240502 20:45:59.368721 23090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.395323 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:59.407470 23131 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 2 index: 105. (index mismatch)
I20240502 20:45:59.407953 25352 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 20:45:59.411310 23400 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 2 index: 105. (index mismatch)
I20240502 20:45:59.411988 25352 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 20:45:59.420869 24823 consensus_queue.cc:237] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 2.105, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:45:59.423005 23131 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch)
W20240502 20:45:59.423218 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:59.428579 25352 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 20:45:59.434108 25265 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.106: config changed from index 94 to 106, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:59.435992 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b80c0d7e5de4ad58511694ec615dfef with cas_config_opid_index 94: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:59.435863 23131 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Committing config change with OpId 2.106: config changed from index 94 to 106, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:45:59.437565 22985 catalog_manager.cc:5484] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 94 to 106, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:59.442570 23382 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b80c0d7e5de4ad58511694ec615dfef with delete_type TABLET_DATA_TOMBSTONED (TS ee6fe9bcf23a450cb5432a8205660598 not found in new config with opid_index 106) from {username='slave'} at 127.0.0.1:39856
I20240502 20:45:59.443197 25449 tablet_replica.cc:330] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: stopping tablet replica
I20240502 20:45:59.443414 25449 raft_consensus.cc:2229] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:59.445318 25449 raft_consensus.cc:2258] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:59.446228 25449 ts_tablet_manager.cc:1875] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:59.449370 25449 ts_tablet_manager.cc:1888] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.105
I20240502 20:45:59.449508 25449 log.cc:1196] T 4b80c0d7e5de4ad58511694ec615dfef P ee6fe9bcf23a450cb5432a8205660598: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/4b80c0d7e5de4ad58511694ec615dfef
I20240502 20:45:59.450248 22971 catalog_manager.cc:4830] TS ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261): tablet 4b80c0d7e5de4ad58511694ec615dfef (table rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]) successfully deleted
W20240502 20:45:59.459081 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.490482 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
W20240502 20:45:59.524106 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Service unavailable: leader transfer in progress
I20240502 20:45:59.526798 24967 raft_consensus.cc:979] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: : Instructing follower 3db3750c037e4393927784c55f66d349 to start an election
I20240502 20:45:59.526960 24967 raft_consensus.cc:1067] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Signalling peer 3db3750c037e4393927784c55f66d349 to start an election
I20240502 20:45:59.527608 23267 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64"
dest_uuid: "3db3750c037e4393927784c55f66d349"
 from {username='slave'} at 127.19.66.129:40495
I20240502 20:45:59.527765 23267 raft_consensus.cc:482] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:45:59.527812 23267 raft_consensus.cc:3037] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:59.528995 23267 raft_consensus.cc:504] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:59.529531 23267 leader_election.cc:288] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 2 election: Requested vote from peers 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439), c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861)
I20240502 20:45:59.530330 23130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 2 candidate_status { last_received { term: 1 index: 72 } } ignore_live_leader: true dest_uuid: "140dd1ea33be47e99530c773c5439d56"
I20240502 20:45:59.530339 24689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64" candidate_uuid: "3db3750c037e4393927784c55f66d349" candidate_term: 2 candidate_status { last_received { term: 1 index: 72 } } ignore_live_leader: true dest_uuid: "c9af0b94bfe2492bb34c7255879a7328"
I20240502 20:45:59.530483 23130 raft_consensus.cc:3032] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:45:59.530543 23130 raft_consensus.cc:729] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 140dd1ea33be47e99530c773c5439d56, State: Running, Role: LEADER
I20240502 20:45:59.530607 24689 raft_consensus.cc:3037] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:59.530727 23130 consensus_queue.cc:260] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 72, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:59.530953 23130 raft_consensus.cc:3037] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:45:59.531951 24689 raft_consensus.cc:2454] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 2.
I20240502 20:45:59.531980 23130 raft_consensus.cc:2454] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db3750c037e4393927784c55f66d349 in term 2.
I20240502 20:45:59.532843 23202 leader_election.cc:302] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db3750c037e4393927784c55f66d349, c9af0b94bfe2492bb34c7255879a7328; no voters: 
I20240502 20:45:59.533780 24980 raft_consensus.cc:2781] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:45:59.533897 24980 raft_consensus.cc:686] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 LEADER]: Becoming Leader. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:45:59.534066 24980 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 72, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:59.537336 22985 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 reported cstate change: term changed from 1 to 2, leader changed from 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) to 3db3750c037e4393927784c55f66d349 (127.19.66.130). New cstate: current_term: 2 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 20:45:59.559268 23089 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47736: Illegal state: replica 140dd1ea33be47e99530c773c5439d56 is not leader of this config: current role FOLLOWER
I20240502 20:45:59.593250 23130 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 2 index: 74. (index mismatch)
I20240502 20:45:59.593294 24690 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 2 index: 74. (index mismatch)
I20240502 20:45:59.593789 24910 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240502 20:45:59.596478 24910 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240502 20:45:59.600528 25201 mvcc.cc:205] Tried to move back new op lower bound from 7023340583289327616 to 7023340583053217792. Current Snapshot: MvccSnapshot[applied={T|T < 7023340583289327616}]
I20240502 20:45:59.605746 23267 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 2.74, Last appended by leader: 72, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } }
I20240502 20:45:59.614446 24688 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 2 index: 74. Preceding OpId from leader: term: 2 index: 75. (index mismatch)
I20240502 20:45:59.616487 24980 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240502 20:45:59.621955 24900 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 LEADER]: Committing config change with OpId 2.75: config changed from index 59 to 75, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:59.623991 22972 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet be76bba97022462a86ed4a4cc2a2dd64 with cas_config_opid_index 59: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:45:59.625473 22985 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index 59 to 75, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New cstate: current_term: 2 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:45:59.626716 24688 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Committing config change with OpId 2.75: config changed from index 59 to 75, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } }
I20240502 20:45:59.629122 23113 tablet_service.cc:1514] Processing DeleteTablet for tablet be76bba97022462a86ed4a4cc2a2dd64 with delete_type TABLET_DATA_TOMBSTONED (TS 140dd1ea33be47e99530c773c5439d56 not found in new config with opid_index 75) from {username='slave'} at 127.0.0.1:33346
I20240502 20:45:59.636482 25468 tablet_replica.cc:330] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: stopping tablet replica
I20240502 20:45:59.636790 25468 raft_consensus.cc:2229] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:45:59.636984 25468 pending_rounds.cc:62] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: Trying to abort 2 pending ops.
I20240502 20:45:59.637094 25468 pending_rounds.cc:68] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: Aborting op as it isn't in flight: id { term: 2 index: 74 } timestamp: 7023340583289327616 op_type: WRITE_OP write_request { tablet_id: "be76bba97022462a86ed4a4cc2a2dd64" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000/xo[\274\301{H\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""018b613e4b3a5c79" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023340580680232960 authz_token { token_data: "\010\257\371\317\261\006\"3\n\005slave\022*\n e6d084e24e8342d892514572e869edd1\020\001\030\001 \001(\001" signature: "<redacted>""\2305\214H(\204\340\206M\017V\260\322\366H<\003\034\177p`\331b-r\032d&\304\274\021\010J5\210U\251\374\313VY\246\235\367\201-;\200=\376\336x\001\366 \013n\302\345.\345\341d8" signing_key_seq_num: 0 } } request_id { client_id: "77f39216e3ad4c058856096373ffb073" seq_no: 191 first_incomplete_seq_no: 191 attempt_no: 30 }
I20240502 20:45:59.639108 25468 pending_rounds.cc:68] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: Aborting op as it isn't in flight: id { term: 2 index: 73 } timestamp: 7023340583053217792 op_type: NO_OP noop_request { }
I20240502 20:45:59.639391 25468 raft_consensus.cc:2866] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 20:45:59.639531 25468 raft_consensus.cc:2258] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:45:59.640388 25468 ts_tablet_manager.cc:1875] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:45:59.645330 25468 ts_tablet_manager.cc:1888] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.74
I20240502 20:45:59.646941 25468 log.cc:1196] T be76bba97022462a86ed4a4cc2a2dd64 P 140dd1ea33be47e99530c773c5439d56: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/be76bba97022462a86ed4a4cc2a2dd64
I20240502 20:45:59.649228 22972 catalog_manager.cc:4830] TS 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439): tablet be76bba97022462a86ed4a4cc2a2dd64 (table rebalance_test_table_3 [id=e6d084e24e8342d892514572e869edd1]) successfully deleted
I20240502 20:46:00.163254 24790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:00.169865 23247 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:00.177230 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:00.193670 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:00.210311 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:00.214898 24536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:00.267982 23382 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:00.374689 23267 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 2.94, Last appended by leader: 72, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
W20240502 20:46:00.376364 23199 consensus_peers.cc:485] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 -> Peer 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917): Couldn't send request to peer 09f2f86617004c96b1e186759506cc68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be76bba97022462a86ed4a4cc2a2dd64. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:00.378908 24689 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 2 index: 94. Preceding OpId from leader: term: 2 index: 95. (index mismatch)
I20240502 20:46:00.379699 24038 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20240502 20:46:00.402261 24980 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 LEADER]: Committing config change with OpId 2.95: config changed from index 75 to 95, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } } }
I20240502 20:46:00.406117 22982 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index 75 to 95, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New cstate: current_term: 2 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:00.408952 24689 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Committing config change with OpId 2.95: config changed from index 75 to 95, NON_VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } } }
I20240502 20:46:00.452587 23400 consensus_queue.cc:237] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 2.142, Last appended by leader: 86, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } }
I20240502 20:46:00.459458 24554 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 2 index: 142. Preceding OpId from leader: term: 2 index: 143. (index mismatch)
I20240502 20:46:00.460007 24995 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I20240502 20:46:00.464413 24995 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Committing config change with OpId 2.143: config changed from index 89 to 143, NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) added. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } } }
I20240502 20:46:00.466449 24554 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Committing config change with OpId 2.143: config changed from index 89 to 143, NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) added. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } } }
W20240502 20:46:00.471019 23336 consensus_peers.cc:485] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 -> Peer 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132:45001): Couldn't send request to peer 94ac80c7dd7f4767bb357b89c547d8b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37ab508cf18c46d8ad1775086354f379. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:00.479686 22982 catalog_manager.cc:5484] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: config changed from index 89 to 143, NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) added. New cstate: current_term: 2 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:00.571039 24824 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 2.93, Last appended by leader: 47, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:46:00.582877 23130 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 93. Preceding OpId from leader: term: 2 index: 94. (index mismatch)
I20240502 20:46:00.594516 25446 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240502 20:46:00.604614 25265 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.94: config changed from index 52 to 94, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } } }
I20240502 20:46:00.607924 23130 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Committing config change with OpId 2.94: config changed from index 52 to 94, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } } }
I20240502 20:46:00.607924 22982 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 52 to 94, NON_VOTER b30503059718458bb016185550f5e0fb (127.19.66.133) added. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:00.634435 24754 consensus_peers.cc:485] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 -> Peer b30503059718458bb016185550f5e0fb (127.19.66.133:45729): Couldn't send request to peer b30503059718458bb016185550f5e0fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f6c6fc06ec341e28a4b9f231d78ac30. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:00.643790 24824 consensus_queue.cc:237] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 134, Committed index: 134, Last appended: 2.134, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } }
W20240502 20:46:00.646286 24757 consensus_peers.cc:485] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 -> Peer 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132:45001): Couldn't send request to peer 94ac80c7dd7f4767bb357b89c547d8b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b80c0d7e5de4ad58511694ec615dfef. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:00.649257 23131 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 134. Preceding OpId from leader: term: 2 index: 135. (index mismatch)
I20240502 20:46:00.659461 25265 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240502 20:46:00.663208 25446 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.135: config changed from index 106 to 135, NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) added. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } } }
I20240502 20:46:00.663743 23131 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Committing config change with OpId 2.135: config changed from index 106 to 135, NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) added. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } } }
I20240502 20:46:00.666787 22987 catalog_manager.cc:5484] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 106 to 135, NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) added. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:00.771759 25577 ts_tablet_manager.cc:902] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: Initiating tablet copy from peer 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:46:00.772473 25577 tablet_copy_client.cc:308] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:35415
I20240502 20:46:00.775799 23289 tablet_copy_service.cc:140] P 3db3750c037e4393927784c55f66d349: Received BeginTabletCopySession request for tablet be76bba97022462a86ed4a4cc2a2dd64 from peer 09f2f86617004c96b1e186759506cc68 ({username='slave'} at 127.19.66.135:38529)
I20240502 20:46:00.775945 23289 tablet_copy_service.cc:161] P 3db3750c037e4393927784c55f66d349: Beginning new tablet copy session on tablet be76bba97022462a86ed4a4cc2a2dd64 from peer 09f2f86617004c96b1e186759506cc68 at {username='slave'} at 127.19.66.135:38529: session id = 09f2f86617004c96b1e186759506cc68-be76bba97022462a86ed4a4cc2a2dd64
I20240502 20:46:00.777050 23289 tablet_copy_source_session.cc:205] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:00.781289 25577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be76bba97022462a86ed4a4cc2a2dd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:00.785105 25577 tablet_copy_client.cc:783] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:00.785514 25577 tablet_copy_client.cc:647] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:00.802058 25577 tablet_copy_client.cc:523] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:00.803762 25577 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: Bootstrap starting.
I20240502 20:46:01.017241 25590 ts_tablet_manager.cc:902] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb: Initiating tablet copy from peer 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917)
I20240502 20:46:01.018028 25590 tablet_copy_client.cc:308] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.135:35917
I20240502 20:46:01.035671 25577 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=0 applied=99 ignored=0} inserts{seen=780 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:01.036759 25577 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: Bootstrap complete.
I20240502 20:46:01.036968 25577 ts_tablet_manager.cc:1367] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: Time spent bootstrapping tablet: real 0.233s	user 0.095s	sys 0.016s
I20240502 20:46:01.037644 25577 raft_consensus.cc:348] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
I20240502 20:46:01.037791 25577 raft_consensus.cc:729] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Initialized, Role: LEARNER
I20240502 20:46:01.038089 25577 consensus_queue.cc:260] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 99, Last appended: 2.99, Last appended by leader: 99, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: true } }
I20240502 20:46:01.042528 25577 ts_tablet_manager.cc:1398] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:46:01.048700 23289 tablet_copy_service.cc:342] P 3db3750c037e4393927784c55f66d349: Request end of tablet copy session 09f2f86617004c96b1e186759506cc68-be76bba97022462a86ed4a4cc2a2dd64 received from {username='slave'} at 127.19.66.135:38529
I20240502 20:46:01.048828 23289 tablet_copy_service.cc:433] P 3db3750c037e4393927784c55f66d349: ending tablet copy session 09f2f86617004c96b1e186759506cc68-be76bba97022462a86ed4a4cc2a2dd64 on tablet be76bba97022462a86ed4a4cc2a2dd64 with peer 09f2f86617004c96b1e186759506cc68
I20240502 20:46:01.051875 25603 ts_tablet_manager.cc:902] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2: Initiating tablet copy from peer ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:46:01.052531 25603 tablet_copy_client.cc:308] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:39261
I20240502 20:46:01.069651 23424 tablet_copy_service.cc:140] P ee6fe9bcf23a450cb5432a8205660598: Received BeginTabletCopySession request for tablet 37ab508cf18c46d8ad1775086354f379 from peer 94ac80c7dd7f4767bb357b89c547d8b2 ({username='slave'} at 127.19.66.132:39099)
I20240502 20:46:01.069883 23424 tablet_copy_service.cc:161] P ee6fe9bcf23a450cb5432a8205660598: Beginning new tablet copy session on tablet 37ab508cf18c46d8ad1775086354f379 from peer 94ac80c7dd7f4767bb357b89c547d8b2 at {username='slave'} at 127.19.66.132:39099: session id = 94ac80c7dd7f4767bb357b89c547d8b2-37ab508cf18c46d8ad1775086354f379
I20240502 20:46:01.072331 23424 tablet_copy_source_session.cc:205] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:01.074178 25603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab508cf18c46d8ad1775086354f379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.076776 25603 tablet_copy_client.cc:783] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:01.076989 25603 tablet_copy_client.cc:647] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:01.082149 24844 tablet_copy_service.cc:140] P 09f2f86617004c96b1e186759506cc68: Received BeginTabletCopySession request for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 from peer b30503059718458bb016185550f5e0fb ({username='slave'} at 127.19.66.133:46033)
I20240502 20:46:01.082324 24844 tablet_copy_service.cc:161] P 09f2f86617004c96b1e186759506cc68: Beginning new tablet copy session on tablet 0f6c6fc06ec341e28a4b9f231d78ac30 from peer b30503059718458bb016185550f5e0fb at {username='slave'} at 127.19.66.133:46033: session id = b30503059718458bb016185550f5e0fb-0f6c6fc06ec341e28a4b9f231d78ac30
I20240502 20:46:01.083990 24844 tablet_copy_source_session.cc:205] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:01.084210 25603 tablet_copy_client.cc:523] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:01.085820 25590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6c6fc06ec341e28a4b9f231d78ac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.088413 25590 tablet_copy_client.cc:783] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:01.089550 25590 tablet_copy_client.cc:647] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:01.103999 25603 tablet_bootstrap.cc:492] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap starting.
I20240502 20:46:01.105960 25590 tablet_copy_client.cc:523] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:01.108906 25590 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb: Bootstrap starting.
I20240502 20:46:01.228529 25626 ts_tablet_manager.cc:902] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2: Initiating tablet copy from peer 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917)
I20240502 20:46:01.230345 25626 tablet_copy_client.cc:308] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.135:35917
I20240502 20:46:01.231582 24844 tablet_copy_service.cc:140] P 09f2f86617004c96b1e186759506cc68: Received BeginTabletCopySession request for tablet 4b80c0d7e5de4ad58511694ec615dfef from peer 94ac80c7dd7f4767bb357b89c547d8b2 ({username='slave'} at 127.19.66.132:39645)
I20240502 20:46:01.231714 24844 tablet_copy_service.cc:161] P 09f2f86617004c96b1e186759506cc68: Beginning new tablet copy session on tablet 4b80c0d7e5de4ad58511694ec615dfef from peer 94ac80c7dd7f4767bb357b89c547d8b2 at {username='slave'} at 127.19.66.132:39645: session id = 94ac80c7dd7f4767bb357b89c547d8b2-4b80c0d7e5de4ad58511694ec615dfef
I20240502 20:46:01.232710 24844 tablet_copy_source_session.cc:205] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:01.248778 25626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b80c0d7e5de4ad58511694ec615dfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:01.313091 25590 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=112 overwritten=0 applied=112 ignored=0} inserts{seen=793 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:01.314040 25590 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb: Bootstrap complete.
I20240502 20:46:01.314317 25590 ts_tablet_manager.cc:1367] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb: Time spent bootstrapping tablet: real 0.205s	user 0.108s	sys 0.010s
I20240502 20:46:01.314801 25590 raft_consensus.cc:348] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:46:01.314941 25590 raft_consensus.cc:729] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b30503059718458bb016185550f5e0fb, State: Initialized, Role: LEARNER
I20240502 20:46:01.315101 25590 consensus_queue.cc:260] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 112, Last appended: 2.112, Last appended by leader: 112, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: true } }
I20240502 20:46:01.315490 25590 ts_tablet_manager.cc:1398] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:01.316110 24844 tablet_copy_service.cc:342] P 09f2f86617004c96b1e186759506cc68: Request end of tablet copy session b30503059718458bb016185550f5e0fb-0f6c6fc06ec341e28a4b9f231d78ac30 received from {username='slave'} at 127.19.66.133:46033
I20240502 20:46:01.316234 24844 tablet_copy_service.cc:433] P 09f2f86617004c96b1e186759506cc68: ending tablet copy session b30503059718458bb016185550f5e0fb-0f6c6fc06ec341e28a4b9f231d78ac30 on tablet 0f6c6fc06ec341e28a4b9f231d78ac30 with peer b30503059718458bb016185550f5e0fb
I20240502 20:46:01.324928 25626 tablet_copy_client.cc:783] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:01.325121 25626 tablet_copy_client.cc:647] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:01.337867 25626 tablet_copy_client.cc:523] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:01.340168 24823 raft_consensus.cc:1203] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.94->[2.95-2.112]   Dedup: 2.99->[2.100-2.112]
I20240502 20:46:01.354568 25626 tablet_bootstrap.cc:492] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap starting.
I20240502 20:46:01.412767 25603 tablet_bootstrap.cc:492] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=160 overwritten=0 applied=160 ignored=0} inserts{seen=828 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:01.413151 25603 tablet_bootstrap.cc:492] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap complete.
I20240502 20:46:01.413380 25603 ts_tablet_manager.cc:1367] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2: Time spent bootstrapping tablet: real 0.309s	user 0.154s	sys 0.004s
I20240502 20:46:01.413911 25603 raft_consensus.cc:348] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } }
I20240502 20:46:01.414047 25603 raft_consensus.cc:729] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 94ac80c7dd7f4767bb357b89c547d8b2, State: Initialized, Role: LEARNER
I20240502 20:46:01.414150 25603 consensus_queue.cc:260] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 160, Last appended: 2.160, Last appended by leader: 160, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } }
I20240502 20:46:01.414573 25603 ts_tablet_manager.cc:1398] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:01.417616 23424 tablet_copy_service.cc:342] P ee6fe9bcf23a450cb5432a8205660598: Request end of tablet copy session 94ac80c7dd7f4767bb357b89c547d8b2-37ab508cf18c46d8ad1775086354f379 received from {username='slave'} at 127.19.66.132:39099
I20240502 20:46:01.417737 23424 tablet_copy_service.cc:433] P ee6fe9bcf23a450cb5432a8205660598: ending tablet copy session 94ac80c7dd7f4767bb357b89c547d8b2-37ab508cf18c46d8ad1775086354f379 on tablet 37ab508cf18c46d8ad1775086354f379 with peer 94ac80c7dd7f4767bb357b89c547d8b2
I20240502 20:46:01.430425 24900 raft_consensus.cc:1050] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: attempting to promote NON_VOTER 09f2f86617004c96b1e186759506cc68 to VOTER
I20240502 20:46:01.430887 24900 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 2.113, Last appended by leader: 72, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:46:01.435576 24824 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 2 LEARNER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 2 index: 113. Preceding OpId from leader: term: 2 index: 114. (index mismatch)
I20240502 20:46:01.436025 24688 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 2 index: 113. Preceding OpId from leader: term: 2 index: 114. (index mismatch)
I20240502 20:46:01.436560 24980 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20240502 20:46:01.438597 25095 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20240502 20:46:01.442816 25095 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 LEADER]: Committing config change with OpId 2.114: config changed from index 95 to 114, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:46:01.443492 24824 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Committing config change with OpId 2.114: config changed from index 95 to 114, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:46:01.446345 22987 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index 95 to 114, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:01.470258 24687 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Committing config change with OpId 2.114: config changed from index 95 to 114, 09f2f86617004c96b1e186759506cc68 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:46:01.515199 24554 raft_consensus.cc:1203] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.93->[2.94-2.121]   Dedup: 2.112->[2.113-2.121]
I20240502 20:46:01.550159 25626 tablet_bootstrap.cc:492] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=150 overwritten=0 applied=150 ignored=0} inserts{seen=777 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:01.550561 25626 tablet_bootstrap.cc:492] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2: Bootstrap complete.
I20240502 20:46:01.550807 25626 ts_tablet_manager.cc:1367] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2: Time spent bootstrapping tablet: real 0.196s	user 0.138s	sys 0.005s
I20240502 20:46:01.551326 25626 raft_consensus.cc:348] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } }
I20240502 20:46:01.551474 25626 raft_consensus.cc:729] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 94ac80c7dd7f4767bb357b89c547d8b2, State: Initialized, Role: LEARNER
I20240502 20:46:01.551604 25626 consensus_queue.cc:260] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 150, Last appended: 2.150, Last appended by leader: 150, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: true } }
I20240502 20:46:01.551913 25626 ts_tablet_manager.cc:1398] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:01.553022 24844 tablet_copy_service.cc:342] P 09f2f86617004c96b1e186759506cc68: Request end of tablet copy session 94ac80c7dd7f4767bb357b89c547d8b2-4b80c0d7e5de4ad58511694ec615dfef received from {username='slave'} at 127.19.66.132:39645
I20240502 20:46:01.553213 24844 tablet_copy_service.cc:433] P 09f2f86617004c96b1e186759506cc68: ending tablet copy session 94ac80c7dd7f4767bb357b89c547d8b2-4b80c0d7e5de4ad58511694ec615dfef on tablet 4b80c0d7e5de4ad58511694ec615dfef with peer 94ac80c7dd7f4767bb357b89c547d8b2
I20240502 20:46:01.603070 25553 raft_consensus.cc:1050] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: attempting to promote NON_VOTER b30503059718458bb016185550f5e0fb to VOTER
I20240502 20:46:01.603565 25553 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 2.122, Last appended by leader: 47, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
I20240502 20:46:01.605696 24554 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 LEARNER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 122. Preceding OpId from leader: term: 2 index: 123. (index mismatch)
I20240502 20:46:01.606066 23131 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 121. Preceding OpId from leader: term: 2 index: 123. (index mismatch)
I20240502 20:46:01.606122 25446 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I20240502 20:46:01.608832 24422 raft_consensus.cc:1203] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.142->[2.143-2.173]   Dedup: 2.160->[2.161-2.173]
I20240502 20:46:01.612233 25446 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.123: config changed from index 94 to 123, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:46:01.620115 24554 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Committing config change with OpId 2.123: config changed from index 94 to 123, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:46:01.622237 25265 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I20240502 20:46:01.623819 22982 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb reported cstate change: config changed from index 94 to 123, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:46:01.629033 24823 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 2.124, Last appended by leader: 47, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } }
I20240502 20:46:01.630863 24554 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 124. Preceding OpId from leader: term: 2 index: 125. (index mismatch)
I20240502 20:46:01.631331 25553 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 20:46:01.633203 25265 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.125: config changed from index 123 to 125, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:46:01.635327 24553 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Committing config change with OpId 2.125: config changed from index 123 to 125, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:46:01.635577 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 with cas_config_opid_index 123: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:01.641388 23130 raft_consensus.cc:1203] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.121->[2.122-2.123]   Dedup: 2.122->[2.123-2.123]
I20240502 20:46:01.641973 23130 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Committing config change with OpId 2.123: config changed from index 94 to 123, b30503059718458bb016185550f5e0fb (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:46:01.642503 22982 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb reported cstate change: config changed from index 123 to 125, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } }
I20240502 20:46:01.645762 23113 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 with delete_type TABLET_DATA_TOMBSTONED (TS 140dd1ea33be47e99530c773c5439d56 not found in new config with opid_index 125) from {username='slave'} at 127.0.0.1:33346
I20240502 20:46:01.646342 25680 tablet_replica.cc:330] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56: stopping tablet replica
I20240502 20:46:01.646576 25680 raft_consensus.cc:2229] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:01.646816 25680 raft_consensus.cc:2258] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:01.647723 25680 ts_tablet_manager.cc:1875] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:01.652707 25680 ts_tablet_manager.cc:1888] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.123
I20240502 20:46:01.652920 25680 log.cc:1196] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 140dd1ea33be47e99530c773c5439d56: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/0f6c6fc06ec341e28a4b9f231d78ac30
I20240502 20:46:01.653661 22972 catalog_manager.cc:4830] TS 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439): tablet 0f6c6fc06ec341e28a4b9f231d78ac30 (table rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]) successfully deleted
I20240502 20:46:01.654424 23113 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 123 (current committed config index is 125)) from {username='slave'} at 127.0.0.1:33346
I20240502 20:46:01.655332 22972 catalog_manager.cc:4830] TS 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439): tablet 0f6c6fc06ec341e28a4b9f231d78ac30 (table rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]) successfully deleted
I20240502 20:46:01.673981 23267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64"
dest_uuid: "3db3750c037e4393927784c55f66d349"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:42996
I20240502 20:46:01.674149 23267 raft_consensus.cc:595] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 LEADER]: Received request to transfer leadership
W20240502 20:46:01.674319 25589 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:01.676864 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:01.682518 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.684327 24995 raft_consensus.cc:1050] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: attempting to promote NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 to VOTER
I20240502 20:46:01.684703 24995 consensus_queue.cc:237] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 2.175, Last appended by leader: 86, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } }
I20240502 20:46:01.686463 24422 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 LEARNER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 2 index: 175. Preceding OpId from leader: term: 2 index: 176. (index mismatch)
I20240502 20:46:01.686614 24554 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 2 index: 175. Preceding OpId from leader: term: 2 index: 176. (index mismatch)
I20240502 20:46:01.686983 25068 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s
I20240502 20:46:01.687712 24994 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s
I20240502 20:46:01.688771 25068 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Committing config change with OpId 2.176: config changed from index 143 to 176, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
W20240502 20:46:01.689059 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.689273 24422 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 FOLLOWER]: Committing config change with OpId 2.176: config changed from index 143 to 176, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:46:01.692026 22978 catalog_manager.cc:5484] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 reported cstate change: config changed from index 143 to 176, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
W20240502 20:46:01.695964 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.703298 24556 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Committing config change with OpId 2.176: config changed from index 143 to 176, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
W20240502 20:46:01.706723 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.714570 24422 raft_consensus.cc:1203] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.134->[2.135-2.163]   Dedup: 2.150->[2.151-2.163]
W20240502 20:46:01.715446 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:01.732749 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:01.745311 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.748296 25553 raft_consensus.cc:1050] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68: attempting to promote NON_VOTER 94ac80c7dd7f4767bb357b89c547d8b2 to VOTER
I20240502 20:46:01.748695 25553 consensus_queue.cc:237] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 164, Committed index: 164, Last appended: 2.164, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } }
I20240502 20:46:01.750109 23130 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 164. Preceding OpId from leader: term: 2 index: 165. (index mismatch)
I20240502 20:46:01.750198 24421 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 LEARNER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 164. Preceding OpId from leader: term: 2 index: 165. (index mismatch)
I20240502 20:46:01.750648 25553 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240502 20:46:01.751802 25553 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 163, Time since last communication: 0.000s
I20240502 20:46:01.753002 25560 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.165: config changed from index 135 to 165, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:46:01.755700 23130 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Committing config change with OpId 2.165: config changed from index 135 to 165, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:46:01.756280 22982 catalog_manager.cc:5484] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 135 to 165, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:01.767724 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.769125 24422 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 FOLLOWER]: Committing config change with OpId 2.165: config changed from index 135 to 165, 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "140dd1ea33be47e99530c773c5439d56" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 45439 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:46:01.773926 24823 consensus_queue.cc:237] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 165, Committed index: 165, Last appended: 2.166, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } }
W20240502 20:46:01.780419 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.786751 24421 raft_consensus.cc:1261] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 166. Preceding OpId from leader: term: 2 index: 167. (index mismatch)
I20240502 20:46:01.787693 25553 consensus_queue.cc:1035] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 165, Time since last communication: 0.000s
W20240502 20:46:01.793918 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.798086 25553 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.167: config changed from index 165 to 167, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:46:01.798573 24422 raft_consensus.cc:2932] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 FOLLOWER]: Committing config change with OpId 2.167: config changed from index 165 to 167, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:46:01.801343 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b80c0d7e5de4ad58511694ec615dfef with cas_config_opid_index 165: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:01.801848 22978 catalog_manager.cc:5484] T 4b80c0d7e5de4ad58511694ec615dfef P 94ac80c7dd7f4767bb357b89c547d8b2 reported cstate change: config changed from index 165 to 167, VOTER 140dd1ea33be47e99530c773c5439d56 (127.19.66.129) evicted. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
W20240502 20:46:01.810328 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.813508 23113 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b80c0d7e5de4ad58511694ec615dfef with delete_type TABLET_DATA_TOMBSTONED (TS 140dd1ea33be47e99530c773c5439d56 not found in new config with opid_index 167) from {username='slave'} at 127.0.0.1:33346
I20240502 20:46:01.814684 25680 tablet_replica.cc:330] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: stopping tablet replica
I20240502 20:46:01.814855 25680 raft_consensus.cc:2229] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:01.814996 25680 pending_rounds.cc:62] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: Trying to abort 1 pending ops.
I20240502 20:46:01.815035 25680 pending_rounds.cc:68] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: Aborting op as it isn't in flight: id { term: 2 index: 166 } timestamp: 7023340592186212352 op_type: WRITE_OP write_request { tablet_id: "4b80c0d7e5de4ad58511694ec615dfef" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\036\370\214xs\302\333E\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a9a14012b2eb27d7" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023340592140947456 authz_token { token_data: "\010\257\371\317\261\006\"3\n\005slave\022*\n ceec311346a6484f9413d765ea4f9f32\020\001\030\001 \001(\001" signature: "<redacted>"")\017)\240\3433|\013\t\275sA\026\217P\336 /\0324\321\367{\341\274&hb\037\341/I\337hNK\266\325\241P\233\331\375Pk9\342H\300t\360\3674\223\022\007\276\212\347\353\237\314\333\340" signing_key_seq_num: 0 } } request_id { client_id: "c48dc28bf7f94cf7a3ae39f653c109b3" seq_no: 399 first_incomplete_seq_no: 399 attempt_no: 0 }
I20240502 20:46:01.815392 25680 raft_consensus.cc:2258] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:01.818368 25680 ts_tablet_manager.cc:1875] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 20:46:01.825613 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.830718 25680 ts_tablet_manager.cc:1888] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.166
I20240502 20:46:01.830904 25680 log.cc:1196] T 4b80c0d7e5de4ad58511694ec615dfef P 140dd1ea33be47e99530c773c5439d56: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/4b80c0d7e5de4ad58511694ec615dfef
I20240502 20:46:01.831869 22972 catalog_manager.cc:4830] TS 140dd1ea33be47e99530c773c5439d56 (127.19.66.129:45439): tablet 4b80c0d7e5de4ad58511694ec615dfef (table rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]) successfully deleted
W20240502 20:46:01.843340 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:01.861346 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:01.883556 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:01.905601 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:01.934083 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.949934 23398 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379"
dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33202
I20240502 20:46:01.950104 23398 raft_consensus.cc:595] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Received request to transfer leadership
W20240502 20:46:01.963454 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:01.987936 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:01.993518 23267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64"
dest_uuid: "3db3750c037e4393927784c55f66d349"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43010
I20240502 20:46:01.993718 23267 raft_consensus.cc:595] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 LEADER]: Received request to transfer leadership
W20240502 20:46:02.012835 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:02.013494 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.019142 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.024948 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.032202 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.038349 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.039669 23227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:02.042685 23398 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379"
dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33218
I20240502 20:46:02.042871 23398 raft_consensus.cc:595] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Received request to transfer leadership
W20240502 20:46:02.046559 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.057919 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.068717 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:02.069950 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.080868 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
I20240502 20:46:02.082051 23267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64"
dest_uuid: "3db3750c037e4393927784c55f66d349"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43028
I20240502 20:46:02.082218 23267 raft_consensus.cc:595] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 LEADER]: Received request to transfer leadership
W20240502 20:46:02.095142 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.097681 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:02.108459 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.121714 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.125290 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
W20240502 20:46:02.141527 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.154304 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Service unavailable: leader transfer in progress
I20240502 20:46:02.159149 24038 raft_consensus.cc:979] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: : Instructing follower 09f2f86617004c96b1e186759506cc68 to start an election
I20240502 20:46:02.159303 24038 raft_consensus.cc:1067] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 LEADER]: Signalling peer 09f2f86617004c96b1e186759506cc68 to start an election
W20240502 20:46:02.159551 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
I20240502 20:46:02.159791 24823 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64"
dest_uuid: "09f2f86617004c96b1e186759506cc68"
 from {username='slave'} at 127.19.66.130:38299
I20240502 20:46:02.159937 24823 raft_consensus.cc:482] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:02.159997 24823 raft_consensus.cc:3037] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:02.161289 24823 raft_consensus.cc:504] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:46:02.161847 24823 leader_election.cc:288] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 3 election: Requested vote from peers 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415), c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861)
I20240502 20:46:02.162649 23267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 3 candidate_status { last_received { term: 2 index: 120 } } ignore_live_leader: true dest_uuid: "3db3750c037e4393927784c55f66d349"
I20240502 20:46:02.162786 23267 raft_consensus.cc:3032] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:46:02.162832 23267 raft_consensus.cc:729] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:46:02.162973 23267 consensus_queue.cc:260] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 2.120, Last appended by leader: 120, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:46:02.163147 23267 raft_consensus.cc:3037] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:02.164285 23267 raft_consensus.cc:2454] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 3.
I20240502 20:46:02.164745 24755 leader_election.cc:302] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f2f86617004c96b1e186759506cc68, 3db3750c037e4393927784c55f66d349; no voters: 
I20240502 20:46:02.165022 25446 raft_consensus.cc:2781] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:46:02.165148 25446 raft_consensus.cc:686] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 3 LEADER]: Becoming Leader. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Running, Role: LEADER
I20240502 20:46:02.165302 25446 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 2.120, Last appended by leader: 120, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:46:02.167918 22982 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 reported cstate change: term changed from 2 to 3, leader changed from 3db3750c037e4393927784c55f66d349 (127.19.66.130) to 09f2f86617004c96b1e186759506cc68 (127.19.66.135). New cstate: current_term: 3 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:02.173106 24688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be76bba97022462a86ed4a4cc2a2dd64" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 3 candidate_status { last_received { term: 2 index: 120 } } ignore_live_leader: true dest_uuid: "c9af0b94bfe2492bb34c7255879a7328"
I20240502 20:46:02.173285 24688 raft_consensus.cc:3037] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:02.174497 24688 raft_consensus.cc:2454] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 3.
W20240502 20:46:02.179774 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.182749 23226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42764: Illegal state: replica 3db3750c037e4393927784c55f66d349 is not leader of this config: current role FOLLOWER
W20240502 20:46:02.198535 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.217818 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.236717 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
I20240502 20:46:02.257788 23267 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 3 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 120. Preceding OpId from leader: term: 3 index: 122. (index mismatch)
I20240502 20:46:02.258401 25446 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
W20240502 20:46:02.265666 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
I20240502 20:46:02.268013 24688 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 3 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 120. Preceding OpId from leader: term: 3 index: 122. (index mismatch)
I20240502 20:46:02.270305 25560 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
W20240502 20:46:02.289675 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
I20240502 20:46:02.293323 24824 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 3.123, Last appended by leader: 120, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } }
I20240502 20:46:02.296581 24690 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 3 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 3 index: 123. Preceding OpId from leader: term: 3 index: 124. (index mismatch)
I20240502 20:46:02.296988 25446 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 122, Time since last communication: 0.000s
I20240502 20:46:02.302505 25446 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 3 LEADER]: Committing config change with OpId 3.124: config changed from index 114 to 124, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:46:02.305186 24688 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 3 FOLLOWER]: Committing config change with OpId 3.124: config changed from index 114 to 124, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } }
I20240502 20:46:02.305524 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet be76bba97022462a86ed4a4cc2a2dd64 with cas_config_opid_index 114: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:02.306876 22978 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 114 to 124, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New cstate: current_term: 3 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:02.312139 23247 tablet_service.cc:1514] Processing DeleteTablet for tablet be76bba97022462a86ed4a4cc2a2dd64 with delete_type TABLET_DATA_TOMBSTONED (TS 3db3750c037e4393927784c55f66d349 not found in new config with opid_index 124) from {username='slave'} at 127.0.0.1:44302
W20240502 20:46:02.314102 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.340130 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
I20240502 20:46:02.345364 25750 tablet_replica.cc:330] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: stopping tablet replica
I20240502 20:46:02.345538 25750 raft_consensus.cc:2229] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:02.345652 25750 raft_consensus.cc:2258] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:02.346854 25750 ts_tablet_manager.cc:1875] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:02.352530 25750 ts_tablet_manager.cc:1888] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.123
I20240502 20:46:02.352736 25750 log.cc:1196] T be76bba97022462a86ed4a4cc2a2dd64 P 3db3750c037e4393927784c55f66d349: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/be76bba97022462a86ed4a4cc2a2dd64
I20240502 20:46:02.353631 22972 catalog_manager.cc:4830] TS 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415): tablet be76bba97022462a86ed4a4cc2a2dd64 (table rebalance_test_table_3 [id=e6d084e24e8342d892514572e869edd1]) successfully deleted
I20240502 20:46:02.356232 23398 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379"
dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33232
I20240502 20:46:02.356387 23398 raft_consensus.cc:595] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Received request to transfer leadership
W20240502 20:46:02.367065 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.393107 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
W20240502 20:46:02.420970 23361 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Service unavailable: leader transfer in progress
I20240502 20:46:02.435490 24995 raft_consensus.cc:979] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: : Instructing follower b30503059718458bb016185550f5e0fb to start an election
I20240502 20:46:02.435636 24995 raft_consensus.cc:1067] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Signalling peer b30503059718458bb016185550f5e0fb to start an election
I20240502 20:46:02.436240 24556 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379"
dest_uuid: "b30503059718458bb016185550f5e0fb"
 from {username='slave'} at 127.19.66.131:40113
I20240502 20:46:02.436460 24556 raft_consensus.cc:482] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:02.436532 24556 raft_consensus.cc:3037] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:02.437839 24556 raft_consensus.cc:504] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } }
I20240502 20:46:02.438727 23398 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379"
dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33262
I20240502 20:46:02.438848 23398 raft_consensus.cc:595] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:46:02.438894 24556 leader_election.cc:288] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [CANDIDATE]: Term 3 election: Requested vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261), 94ac80c7dd7f4767bb357b89c547d8b2 (127.19.66.132:45001)
I20240502 20:46:02.439224 23398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379" candidate_uuid: "b30503059718458bb016185550f5e0fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 187 } } ignore_live_leader: true dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
I20240502 20:46:02.439349 23398 raft_consensus.cc:3032] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:46:02.439395 23398 raft_consensus.cc:729] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Running, Role: LEADER
I20240502 20:46:02.439553 23398 consensus_queue.cc:260] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 2.187, Last appended by leader: 187, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } }
I20240502 20:46:02.439805 23398 raft_consensus.cc:3037] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:02.441483 23398 raft_consensus.cc:2454] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30503059718458bb016185550f5e0fb in term 3.
I20240502 20:46:02.442097 24488 leader_election.cc:302] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b30503059718458bb016185550f5e0fb, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:46:02.442572 25774 raft_consensus.cc:2781] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:46:02.443009 25774 raft_consensus.cc:686] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 3 LEADER]: Becoming Leader. State: Replica: b30503059718458bb016185550f5e0fb, State: Running, Role: LEADER
I20240502 20:46:02.444846 25774 consensus_queue.cc:237] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 2.187, Last appended by leader: 187, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } }
I20240502 20:46:02.446049 24420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab508cf18c46d8ad1775086354f379" candidate_uuid: "b30503059718458bb016185550f5e0fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 187 } } ignore_live_leader: true dest_uuid: "94ac80c7dd7f4767bb357b89c547d8b2"
I20240502 20:46:02.446183 24420 raft_consensus.cc:3037] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:02.447345 24420 raft_consensus.cc:2454] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30503059718458bb016185550f5e0fb in term 3.
I20240502 20:46:02.447921 22982 catalog_manager.cc:5484] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb reported cstate change: term changed from 2 to 3, leader changed from ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) to b30503059718458bb016185550f5e0fb (127.19.66.133). New cstate: current_term: 3 leader_uuid: "b30503059718458bb016185550f5e0fb" committed_config { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:02.449796 23360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33006: Illegal state: replica ee6fe9bcf23a450cb5432a8205660598 is not leader of this config: current role FOLLOWER
I20240502 20:46:02.552277 23398 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 3 FOLLOWER]: Refusing update from remote peer b30503059718458bb016185550f5e0fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 187. Preceding OpId from leader: term: 3 index: 189. (index mismatch)
I20240502 20:46:02.552809 25774 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
I20240502 20:46:02.552913 24422 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 3 FOLLOWER]: Refusing update from remote peer b30503059718458bb016185550f5e0fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 187. Preceding OpId from leader: term: 3 index: 189. (index mismatch)
I20240502 20:46:02.555189 25775 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
I20240502 20:46:02.570129 25674 mvcc.cc:205] Tried to move back new op lower bound from 7023340595407622144 to 7023340594978177024. Current Snapshot: MvccSnapshot[applied={T|T < 7023340595407622144}]
I20240502 20:46:02.577807 25690 mvcc.cc:205] Tried to move back new op lower bound from 7023340595407622144 to 7023340594978177024. Current Snapshot: MvccSnapshot[applied={T|T < 7023340595407622144}]
I20240502 20:46:02.607319 24556 consensus_queue.cc:237] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 190, Committed index: 190, Last appended: 3.190, Last appended by leader: 187, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } }
I20240502 20:46:02.621981 24420 raft_consensus.cc:1261] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 3 FOLLOWER]: Refusing update from remote peer b30503059718458bb016185550f5e0fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 190. Preceding OpId from leader: term: 3 index: 191. (index mismatch)
I20240502 20:46:02.623183 25774 consensus_queue.cc:1035] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 191, Last known committed idx: 189, Time since last communication: 0.000s
I20240502 20:46:02.626742 25774 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb [term 3 LEADER]: Committing config change with OpId 3.191: config changed from index 176 to 191, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:46:02.629238 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37ab508cf18c46d8ad1775086354f379 with cas_config_opid_index 176: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:02.631249 22982 catalog_manager.cc:5484] T 37ab508cf18c46d8ad1775086354f379 P b30503059718458bb016185550f5e0fb reported cstate change: config changed from index 176 to 191, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New cstate: current_term: 3 leader_uuid: "b30503059718458bb016185550f5e0fb" committed_config { opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:02.634527 23382 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ab508cf18c46d8ad1775086354f379 with delete_type TABLET_DATA_TOMBSTONED (TS ee6fe9bcf23a450cb5432a8205660598 not found in new config with opid_index 191) from {username='slave'} at 127.0.0.1:39856
I20240502 20:46:02.634995 25788 tablet_replica.cc:330] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: stopping tablet replica
I20240502 20:46:02.635226 25788 raft_consensus.cc:2229] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:02.635449 25788 raft_consensus.cc:2258] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:02.636207 24420 raft_consensus.cc:2932] T 37ab508cf18c46d8ad1775086354f379 P 94ac80c7dd7f4767bb357b89c547d8b2 [term 3 FOLLOWER]: Committing config change with OpId 3.191: config changed from index 176 to 191, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "94ac80c7dd7f4767bb357b89c547d8b2" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 45001 } attrs { promote: false } } }
I20240502 20:46:02.636441 25788 ts_tablet_manager.cc:1875] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:02.639479 25788 ts_tablet_manager.cc:1888] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.190
I20240502 20:46:02.639619 25788 log.cc:1196] T 37ab508cf18c46d8ad1775086354f379 P ee6fe9bcf23a450cb5432a8205660598: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/37ab508cf18c46d8ad1775086354f379
I20240502 20:46:02.640327 22971 catalog_manager.cc:4830] TS ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261): tablet 37ab508cf18c46d8ad1775086354f379 (table rebalance_test_table_0 [id=b45b0775e1d8423585dd5d5a47ec5dbe]) successfully deleted
I20240502 20:46:02.959048 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:02.962826 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:02.974923 24790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:02.990563 24536 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:03.004935 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:03.023468 23247 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:03.073050 23382 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:03.327245 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:03.390281 24790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:03.394394 24536 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:03.404438 23247 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:03.406867 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:03.407435 23382 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:03.452061 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:04.219632 24790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:04.221984 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:04.225312 23247 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:04.231024 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:04.245127 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:04.288502 23382 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:04.293998 24536 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:04.678184 23247 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:04.678891 24790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:04.689306 24536 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:04.689460 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:04.707998 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:04.771042 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:04.789439 23382 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:04.996563 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:05.002864 24536 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:05.003180 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:05.003976 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:05.010064 23247 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:05.011711 23382 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:05.012161 24790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 2aa045eae84748a59afc512c822816b6 | 127.19.66.190:46265 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                 |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                 | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                 | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                    | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                 | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                 | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                 | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                   | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                  | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.19.66.148:41037 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 09f2f86617004c96b1e186759506cc68 | 127.19.66.135:35917 | HEALTHY | <none>   |       4        |       0
 140dd1ea33be47e99530c773c5439d56 | 127.19.66.129:45439 | HEALTHY | <none>   |       3        |       0
 3db3750c037e4393927784c55f66d349 | 127.19.66.130:35415 | HEALTHY | <none>   |       5        |       0
 94ac80c7dd7f4767bb357b89c547d8b2 | 127.19.66.132:45001 | HEALTHY | <none>   |       0        |       0
 b30503059718458bb016185550f5e0fb | 127.19.66.133:45729 | HEALTHY | <none>   |       1        |       0
 c9af0b94bfe2492bb34c7255879a7328 | 127.19.66.134:43861 | HEALTHY | <none>   |       0        |       0
 ee6fe9bcf23a450cb5432a8205660598 | 127.19.66.131:39261 | HEALTHY | <none>   |       3        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                               |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                              | experimental          | all 7 server(s) checked
 local_ip_for_outbound_sockets            | 127.19.66.129                                                                                                                                                    | experimental          | 127.19.66.129:45439
 local_ip_for_outbound_sockets            | 127.19.66.130                                                                                                                                                    | experimental          | 127.19.66.130:35415
 local_ip_for_outbound_sockets            | 127.19.66.131                                                                                                                                                    | experimental          | 127.19.66.131:39261
 local_ip_for_outbound_sockets            | 127.19.66.132                                                                                                                                                    | experimental          | 127.19.66.132:45001
 local_ip_for_outbound_sockets            | 127.19.66.133                                                                                                                                                    | experimental          | 127.19.66.133:45729
 local_ip_for_outbound_sockets            | 127.19.66.134                                                                                                                                                    | experimental          | 127.19.66.134:43861
 local_ip_for_outbound_sockets            | 127.19.66.135                                                                                                                                                    | experimental          | 127.19.66.135:35917
 never_fsync                              | true                                                                                                                                                             | unsafe,advanced       | all 7 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                | unsafe,hidden         | all 7 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                             | experimental,advanced | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                             | unsafe                | all 7 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                               | hidden                | all 7 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.19.66.129:45439
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.19.66.130:35415
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.19.66.131:39261
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.19.66.132:45001
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.19.66.133:45729
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.19.66.134:43861
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.19.66.135:35917

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.19.66.148:41037 | all 7 server(s) checked
 time_source         | builtin             | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 5
 Third Quartile | 5
 Maximum        | 5

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 20:46:05.059019 19722 log_verifier.cc:126] Checking tablet 0f6c6fc06ec341e28a4b9f231d78ac30
I20240502 20:46:05.110110 19722 log_verifier.cc:177] Verified matching terms for 213 ops in tablet 0f6c6fc06ec341e28a4b9f231d78ac30
I20240502 20:46:05.110245 19722 log_verifier.cc:126] Checking tablet 1e226ee7a67b46f687971e7de8099217
I20240502 20:46:05.155007 19722 log_verifier.cc:177] Verified matching terms for 196 ops in tablet 1e226ee7a67b46f687971e7de8099217
I20240502 20:46:05.155135 19722 log_verifier.cc:126] Checking tablet 21b3fcb144894a818b2f649a4473a7ee
I20240502 20:46:05.203094 19722 log_verifier.cc:177] Verified matching terms for 215 ops in tablet 21b3fcb144894a818b2f649a4473a7ee
I20240502 20:46:05.203223 19722 log_verifier.cc:126] Checking tablet 37ab508cf18c46d8ad1775086354f379
I20240502 20:46:05.258508 19722 log_verifier.cc:177] Verified matching terms for 259 ops in tablet 37ab508cf18c46d8ad1775086354f379
I20240502 20:46:05.258656 19722 log_verifier.cc:126] Checking tablet 4b80c0d7e5de4ad58511694ec615dfef
I20240502 20:46:05.316484 19722 log_verifier.cc:177] Verified matching terms for 260 ops in tablet 4b80c0d7e5de4ad58511694ec615dfef
I20240502 20:46:05.316869 19722 log_verifier.cc:126] Checking tablet 4d54cbc40bcc4fa481a04d100ed3c927
I20240502 20:46:05.335285 19722 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 4d54cbc40bcc4fa481a04d100ed3c927
I20240502 20:46:05.335386 19722 log_verifier.cc:126] Checking tablet 5a5fdd7816e345038ed9b6946084a67e
I20240502 20:46:05.395541 19722 log_verifier.cc:177] Verified matching terms for 217 ops in tablet 5a5fdd7816e345038ed9b6946084a67e
I20240502 20:46:05.395696 19722 log_verifier.cc:126] Checking tablet 711786372ea5495abebec7086f958fef
I20240502 20:46:05.448387 19722 log_verifier.cc:177] Verified matching terms for 231 ops in tablet 711786372ea5495abebec7086f958fef
I20240502 20:46:05.448510 19722 log_verifier.cc:126] Checking tablet 798923743c9d444f917b747309d2285c
I20240502 20:46:05.504199 19722 log_verifier.cc:177] Verified matching terms for 252 ops in tablet 798923743c9d444f917b747309d2285c
I20240502 20:46:05.504323 19722 log_verifier.cc:126] Checking tablet 96ee320aaf2c40a9885f934c5c653c78
I20240502 20:46:05.572659 19722 log_verifier.cc:177] Verified matching terms for 270 ops in tablet 96ee320aaf2c40a9885f934c5c653c78
I20240502 20:46:05.572821 19722 log_verifier.cc:126] Checking tablet 98e34e4543d24b46963049828877d3d5
I20240502 20:46:05.622329 19722 log_verifier.cc:177] Verified matching terms for 189 ops in tablet 98e34e4543d24b46963049828877d3d5
I20240502 20:46:05.622447 19722 log_verifier.cc:126] Checking tablet b0f06fbb9f58417793a6d176b1f1078f
I20240502 20:46:05.678442 19722 log_verifier.cc:177] Verified matching terms for 226 ops in tablet b0f06fbb9f58417793a6d176b1f1078f
I20240502 20:46:05.678575 19722 log_verifier.cc:126] Checking tablet be76bba97022462a86ed4a4cc2a2dd64
I20240502 20:46:05.725927 19722 log_verifier.cc:177] Verified matching terms for 198 ops in tablet be76bba97022462a86ed4a4cc2a2dd64
I20240502 20:46:05.726047 19722 log_verifier.cc:126] Checking tablet ddfa7e208cb1404ba25ebc77a75f7c1a
I20240502 20:46:05.777920 19722 log_verifier.cc:177] Verified matching terms for 240 ops in tablet ddfa7e208cb1404ba25ebc77a75f7c1a
I20240502 20:46:05.778043 19722 log_verifier.cc:126] Checking tablet df89654f87ed43758ba68a6a5582b059
I20240502 20:46:05.837414 19722 log_verifier.cc:177] Verified matching terms for 256 ops in tablet df89654f87ed43758ba68a6a5582b059
I20240502 20:46:05.837742 19722 log_verifier.cc:126] Checking tablet fb9d185d01e44dd49580eeb4ab979346
I20240502 20:46:05.891666 19722 log_verifier.cc:177] Verified matching terms for 215 ops in tablet fb9d185d01e44dd49580eeb4ab979346
I20240502 20:46:05.894234 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.136:0
--local_ip_for_outbound_sockets=127.19.66.136
--webserver_interface=127.19.66.136
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--builtin_ntp_servers=127.19.66.148:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:46:06.023766 26009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:06.024022 26009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:06.024071 26009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:06.027410 26009 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:06.027501 26009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:06.027575 26009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.136
I20240502 20:46:06.032153 26009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.19.66.136
--webserver_port=0
--tserver_master_addrs=127.19.66.190:46265
--never_fsync=true
--heap_profile_path=/tmp/kudu.26009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.136
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:06.033178 26009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:06.034340 26009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:06.044270 26016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:06.044989 26009 server_base.cc:1034] running on GCE node
W20240502 20:46:06.045310 26018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:06.045958 26015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:06.046433 26009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:06.047253 26009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:06.048444 26009 hybrid_clock.cc:648] HybridClock initialized: now 1714682766048387 us; error 70 us; skew 500 ppm
I20240502 20:46:06.050593 26009 webserver.cc:460] Webserver started at http://127.19.66.136:35729/ using document root <none> and password file <none>
I20240502 20:46:06.051187 26009 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:06.051246 26009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:06.051468 26009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:06.053423 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "1469234e70794be0b1c69b7e7ceb219c"
format_stamp: "Formatted at 2024-05-02 20:46:06 on dist-test-slave-gqqg"
I20240502 20:46:06.053953 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "1469234e70794be0b1c69b7e7ceb219c"
format_stamp: "Formatted at 2024-05-02 20:46:06 on dist-test-slave-gqqg"
I20240502 20:46:06.059038 26009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.001s
I20240502 20:46:06.061764 26024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:06.063172 26009 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 20:46:06.063294 26009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1469234e70794be0b1c69b7e7ceb219c"
format_stamp: "Formatted at 2024-05-02 20:46:06 on dist-test-slave-gqqg"
I20240502 20:46:06.063408 26009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:06.097863 26009 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:06.098676 26009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:06.098853 26009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:06.099490 26009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:06.100533 26009 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:06.100582 26009 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:06.100623 26009 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:06.100637 26009 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:06.112964 26009 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.136:37607
I20240502 20:46:06.114516 26009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:46:06.113018 26140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.136:37607 every 8 connection(s)
I20240502 20:46:06.123205 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 26009
I20240502 20:46:06.123359 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 20:46:06.125305 26141 heartbeater.cc:347] Connected to a master server at 127.19.66.190:46265
I20240502 20:46:06.125610 26141 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:06.126240 26141 heartbeater.cc:511] Master 127.19.66.190:46265 requested a full tablet report, sending...
I20240502 20:46:06.127401 22982 ts_manager.cc:194] Registered new tserver with Master: 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136:37607)
I20240502 20:46:06.128054 22982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.136:32919
I20240502 20:46:06.366264 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:06.377153 23247 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:06.377143 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:06.379791 24790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:06.379860 26074 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:06.380869 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:06.385293 24536 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:06.415113 23382 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:06.513315 24790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:06.516223 23247 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 20:46:06.517156 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:06.526154 23382 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:06.538960 24536 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:06.540086 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:06.552887 26074 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:06.570086 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:06.605154 23404 consensus_queue.cc:237] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 1.217, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:06.606714 24689 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
I20240502 20:46:06.607204 26238 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20240502 20:46:06.609413 25905 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Committing config change with OpId 1.218: config changed from index 54 to 218, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } } }
I20240502 20:46:06.611508 24689 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index 54 to 218, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } } }
W20240502 20:46:06.611769 23333 consensus_peers.cc:485] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 -> Peer 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136:37607): Couldn't send request to peer 1469234e70794be0b1c69b7e7ceb219c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a5fdd7816e345038ed9b6946084a67e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.613328 22978 catalog_manager.cc:5484] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 reported cstate change: config changed from index 54 to 218, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.623979 23267 consensus_queue.cc:237] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.256, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:06.626019 24824 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch)
I20240502 20:46:06.626585 26252 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20240502 20:46:06.629057 26252 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.257: config changed from index 73 to 257, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } } }
I20240502 20:46:06.629534 24824 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 73 to 257, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } } }
W20240502 20:46:06.630606 23199 consensus_peers.cc:485] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 -> Peer 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136:37607): Couldn't send request to peer 1469234e70794be0b1c69b7e7ceb219c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df89654f87ed43758ba68a6a5582b059. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.632679 22978 catalog_manager.cc:5484] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index 73 to 257, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.641811 24824 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 198, Committed index: 198, Last appended: 3.198, Last appended by leader: 120, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:06.643419 24689 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 3 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 3 index: 198. Preceding OpId from leader: term: 3 index: 199. (index mismatch)
I20240502 20:46:06.643868 26261 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 199, Last known committed idx: 198, Time since last communication: 0.000s
I20240502 20:46:06.645931 25553 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 3 LEADER]: Committing config change with OpId 3.199: config changed from index 124 to 199, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } } }
I20240502 20:46:06.646366 24689 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 3 FOLLOWER]: Committing config change with OpId 3.199: config changed from index 124 to 199, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } } }
W20240502 20:46:06.647518 24754 consensus_peers.cc:485] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 -> Peer 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136:37607): Couldn't send request to peer 1469234e70794be0b1c69b7e7ceb219c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be76bba97022462a86ed4a4cc2a2dd64. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.649405 22987 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 124 to 199, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New cstate: current_term: 3 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:06.658435 24824 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 2.213, Last appended by leader: 47, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:06.660534 24555 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 213. Preceding OpId from leader: term: 2 index: 214. (index mismatch)
W20240502 20:46:06.660837 24754 consensus_peers.cc:485] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 -> Peer 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136:37607): Couldn't send request to peer 1469234e70794be0b1c69b7e7ceb219c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f6c6fc06ec341e28a4b9f231d78ac30. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:06.661269 25553 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I20240502 20:46:06.663813 26261 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.214: config changed from index 125 to 214, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } } }
I20240502 20:46:06.664417 24555 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Committing config change with OpId 2.214: config changed from index 125 to 214, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } } }
I20240502 20:46:06.667337 22987 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb reported cstate change: config changed from index 125 to 214, NON_VOTER 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) added. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } } }
I20240502 20:46:07.062748 26312 ts_tablet_manager.cc:902] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: Initiating tablet copy from peer ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261)
I20240502 20:46:07.063469 26312 tablet_copy_client.cc:308] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:39261
I20240502 20:46:07.067430 23424 tablet_copy_service.cc:140] P ee6fe9bcf23a450cb5432a8205660598: Received BeginTabletCopySession request for tablet 5a5fdd7816e345038ed9b6946084a67e from peer 1469234e70794be0b1c69b7e7ceb219c ({username='slave'} at 127.19.66.136:47155)
I20240502 20:46:07.067606 23424 tablet_copy_service.cc:161] P ee6fe9bcf23a450cb5432a8205660598: Beginning new tablet copy session on tablet 5a5fdd7816e345038ed9b6946084a67e from peer 1469234e70794be0b1c69b7e7ceb219c at {username='slave'} at 127.19.66.136:47155: session id = 1469234e70794be0b1c69b7e7ceb219c-5a5fdd7816e345038ed9b6946084a67e
I20240502 20:46:07.068481 23424 tablet_copy_source_session.cc:205] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:07.071628 26312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5fdd7816e345038ed9b6946084a67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.076608 26312 tablet_copy_client.cc:783] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:07.076802 26312 tablet_copy_client.cc:647] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.080034 26312 tablet_copy_client.cc:523] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.082078 26312 tablet_bootstrap.cc:492] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap starting.
I20240502 20:46:07.093612 26319 ts_tablet_manager.cc:902] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c: Initiating tablet copy from peer 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917)
I20240502 20:46:07.094753 26319 tablet_copy_client.cc:308] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.135:35917
I20240502 20:46:07.098877 24844 tablet_copy_service.cc:140] P 09f2f86617004c96b1e186759506cc68: Received BeginTabletCopySession request for tablet be76bba97022462a86ed4a4cc2a2dd64 from peer 1469234e70794be0b1c69b7e7ceb219c ({username='slave'} at 127.19.66.136:39395)
I20240502 20:46:07.099015 24844 tablet_copy_service.cc:161] P 09f2f86617004c96b1e186759506cc68: Beginning new tablet copy session on tablet be76bba97022462a86ed4a4cc2a2dd64 from peer 1469234e70794be0b1c69b7e7ceb219c at {username='slave'} at 127.19.66.136:39395: session id = 1469234e70794be0b1c69b7e7ceb219c-be76bba97022462a86ed4a4cc2a2dd64
I20240502 20:46:07.100069 24844 tablet_copy_source_session.cc:205] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:07.101126 26319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be76bba97022462a86ed4a4cc2a2dd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.103721 26319 tablet_copy_client.cc:783] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:07.103888 26319 tablet_copy_client.cc:647] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.105712 26319 tablet_copy_client.cc:523] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.106928 26319 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap starting.
I20240502 20:46:07.108594 26312 log.cc:824] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:07.129226 26327 ts_tablet_manager.cc:902] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c: Initiating tablet copy from peer 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415)
I20240502 20:46:07.130206 26327 tablet_copy_client.cc:308] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:35415
I20240502 20:46:07.130239 26141 heartbeater.cc:503] Master 127.19.66.190:46265 was elected leader, sending a full tablet report...
I20240502 20:46:07.136118 23289 tablet_copy_service.cc:140] P 3db3750c037e4393927784c55f66d349: Received BeginTabletCopySession request for tablet df89654f87ed43758ba68a6a5582b059 from peer 1469234e70794be0b1c69b7e7ceb219c ({username='slave'} at 127.19.66.136:50609)
I20240502 20:46:07.136271 23289 tablet_copy_service.cc:161] P 3db3750c037e4393927784c55f66d349: Beginning new tablet copy session on tablet df89654f87ed43758ba68a6a5582b059 from peer 1469234e70794be0b1c69b7e7ceb219c at {username='slave'} at 127.19.66.136:50609: session id = 1469234e70794be0b1c69b7e7ceb219c-df89654f87ed43758ba68a6a5582b059
I20240502 20:46:07.137398 23289 tablet_copy_source_session.cc:205] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:07.138463 26327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df89654f87ed43758ba68a6a5582b059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.141029 26327 tablet_copy_client.cc:783] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:07.141304 26327 tablet_copy_client.cc:647] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.143889 26327 tablet_copy_client.cc:523] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.145572 26327 tablet_bootstrap.cc:492] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap starting.
I20240502 20:46:07.239871 26353 ts_tablet_manager.cc:902] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c: Initiating tablet copy from peer 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917)
I20240502 20:46:07.240813 26353 tablet_copy_client.cc:308] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.135:35917
I20240502 20:46:07.241642 24844 tablet_copy_service.cc:140] P 09f2f86617004c96b1e186759506cc68: Received BeginTabletCopySession request for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 from peer 1469234e70794be0b1c69b7e7ceb219c ({username='slave'} at 127.19.66.136:39395)
I20240502 20:46:07.241780 24844 tablet_copy_service.cc:161] P 09f2f86617004c96b1e186759506cc68: Beginning new tablet copy session on tablet 0f6c6fc06ec341e28a4b9f231d78ac30 from peer 1469234e70794be0b1c69b7e7ceb219c at {username='slave'} at 127.19.66.136:39395: session id = 1469234e70794be0b1c69b7e7ceb219c-0f6c6fc06ec341e28a4b9f231d78ac30
I20240502 20:46:07.242928 24844 tablet_copy_source_session.cc:205] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:07.244315 26353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6c6fc06ec341e28a4b9f231d78ac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:07.248203 26353 tablet_copy_client.cc:783] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:07.248476 26353 tablet_copy_client.cc:647] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:07.250524 26353 tablet_copy_client.cc:523] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:07.252404 26353 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap starting.
I20240502 20:46:07.268908 26319 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap replayed 1/1 log segments. Stats: ops{read=199 overwritten=0 applied=199 ignored=0} inserts{seen=876 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.269433 26319 tablet_bootstrap.cc:492] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap complete.
I20240502 20:46:07.269860 26319 ts_tablet_manager.cc:1367] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c: Time spent bootstrapping tablet: real 0.163s	user 0.145s	sys 0.012s
I20240502 20:46:07.272933 26319 raft_consensus.cc:348] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:07.273357 26319 raft_consensus.cc:729] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 1469234e70794be0b1c69b7e7ceb219c, State: Initialized, Role: LEARNER
I20240502 20:46:07.273795 26319 consensus_queue.cc:260] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 199, Last appended: 3.199, Last appended by leader: 199, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:07.274892 26319 ts_tablet_manager.cc:1398] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 20:46:07.276331 24844 tablet_copy_service.cc:342] P 09f2f86617004c96b1e186759506cc68: Request end of tablet copy session 1469234e70794be0b1c69b7e7ceb219c-be76bba97022462a86ed4a4cc2a2dd64 received from {username='slave'} at 127.19.66.136:39395
I20240502 20:46:07.276429 24844 tablet_copy_service.cc:433] P 09f2f86617004c96b1e186759506cc68: ending tablet copy session 1469234e70794be0b1c69b7e7ceb219c-be76bba97022462a86ed4a4cc2a2dd64 on tablet be76bba97022462a86ed4a4cc2a2dd64 with peer 1469234e70794be0b1c69b7e7ceb219c
I20240502 20:46:07.284793 26312 tablet_bootstrap.cc:492] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap replayed 1/1 log segments. Stats: ops{read=218 overwritten=0 applied=218 ignored=0} inserts{seen=887 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.285215 26312 tablet_bootstrap.cc:492] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap complete.
I20240502 20:46:07.285512 26312 ts_tablet_manager.cc:1367] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: Time spent bootstrapping tablet: real 0.204s	user 0.182s	sys 0.013s
I20240502 20:46:07.286362 26312 raft_consensus.cc:348] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:07.286516 26312 raft_consensus.cc:729] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1469234e70794be0b1c69b7e7ceb219c, State: Initialized, Role: LEARNER
I20240502 20:46:07.286638 26312 consensus_queue.cc:260] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 218, Last appended: 1.218, Last appended by leader: 218, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:07.286953 26312 ts_tablet_manager.cc:1398] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:07.287864 23424 tablet_copy_service.cc:342] P ee6fe9bcf23a450cb5432a8205660598: Request end of tablet copy session 1469234e70794be0b1c69b7e7ceb219c-5a5fdd7816e345038ed9b6946084a67e received from {username='slave'} at 127.19.66.136:47155
I20240502 20:46:07.287979 23424 tablet_copy_service.cc:433] P ee6fe9bcf23a450cb5432a8205660598: ending tablet copy session 1469234e70794be0b1c69b7e7ceb219c-5a5fdd7816e345038ed9b6946084a67e on tablet 5a5fdd7816e345038ed9b6946084a67e with peer 1469234e70794be0b1c69b7e7ceb219c
I20240502 20:46:07.356997 26327 tablet_bootstrap.cc:492] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap replayed 1/1 log segments. Stats: ops{read=257 overwritten=0 applied=257 ignored=0} inserts{seen=926 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.357421 26327 tablet_bootstrap.cc:492] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap complete.
I20240502 20:46:07.357640 26327 ts_tablet_manager.cc:1367] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c: Time spent bootstrapping tablet: real 0.212s	user 0.182s	sys 0.023s
I20240502 20:46:07.358371 26327 raft_consensus.cc:348] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:07.358594 26327 raft_consensus.cc:729] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1469234e70794be0b1c69b7e7ceb219c, State: Initialized, Role: LEARNER
I20240502 20:46:07.358729 26327 consensus_queue.cc:260] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 257, Last appended: 1.257, Last appended by leader: 257, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:07.359109 26327 ts_tablet_manager.cc:1398] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:07.360378 23289 tablet_copy_service.cc:342] P 3db3750c037e4393927784c55f66d349: Request end of tablet copy session 1469234e70794be0b1c69b7e7ceb219c-df89654f87ed43758ba68a6a5582b059 received from {username='slave'} at 127.19.66.136:50609
I20240502 20:46:07.360487 23289 tablet_copy_service.cc:433] P 3db3750c037e4393927784c55f66d349: ending tablet copy session 1469234e70794be0b1c69b7e7ceb219c-df89654f87ed43758ba68a6a5582b059 on tablet df89654f87ed43758ba68a6a5582b059 with peer 1469234e70794be0b1c69b7e7ceb219c
I20240502 20:46:07.416014 26353 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap replayed 1/1 log segments. Stats: ops{read=214 overwritten=0 applied=214 ignored=0} inserts{seen=892 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:07.416425 26353 tablet_bootstrap.cc:492] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c: Bootstrap complete.
I20240502 20:46:07.416720 26353 ts_tablet_manager.cc:1367] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c: Time spent bootstrapping tablet: real 0.164s	user 0.151s	sys 0.005s
I20240502 20:46:07.417273 26353 raft_consensus.cc:348] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:07.417403 26353 raft_consensus.cc:729] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1469234e70794be0b1c69b7e7ceb219c, State: Initialized, Role: LEARNER
I20240502 20:46:07.417502 26353 consensus_queue.cc:260] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 214, Last appended: 2.214, Last appended by leader: 214, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: true } }
I20240502 20:46:07.417949 26353 ts_tablet_manager.cc:1398] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:07.418612 24844 tablet_copy_service.cc:342] P 09f2f86617004c96b1e186759506cc68: Request end of tablet copy session 1469234e70794be0b1c69b7e7ceb219c-0f6c6fc06ec341e28a4b9f231d78ac30 received from {username='slave'} at 127.19.66.136:39395
I20240502 20:46:07.418715 24844 tablet_copy_service.cc:433] P 09f2f86617004c96b1e186759506cc68: ending tablet copy session 1469234e70794be0b1c69b7e7ceb219c-0f6c6fc06ec341e28a4b9f231d78ac30 on tablet 0f6c6fc06ec341e28a4b9f231d78ac30 with peer 1469234e70794be0b1c69b7e7ceb219c
I20240502 20:46:07.644080 26094 raft_consensus.cc:1203] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.256->[1.257-1.257]   Dedup: 1.257->[]
I20240502 20:46:07.679702 26094 raft_consensus.cc:1203] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.217->[1.218-1.218]   Dedup: 1.218->[]
I20240502 20:46:07.698730 26094 raft_consensus.cc:1203] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c [term 3 LEARNER]: Deduplicated request from leader. Original: 3.198->[3.199-3.199]   Dedup: 3.199->[]
I20240502 20:46:07.835793 26094 raft_consensus.cc:1203] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.213->[2.214-2.214]   Dedup: 2.214->[]
I20240502 20:46:08.023965 26401 raft_consensus.cc:1050] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: attempting to promote NON_VOTER 1469234e70794be0b1c69b7e7ceb219c to VOTER
I20240502 20:46:08.024444 26401 consensus_queue.cc:237] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 1.257, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.026023 24824 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240502 20:46:08.026069 26094 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [term 1 LEARNER]: Refusing update from remote peer 3db3750c037e4393927784c55f66d349: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240502 20:46:08.026558 26429 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240502 20:46:08.026816 25658 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240502 20:46:08.029482 25658 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.029891 24824 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.031121 26094 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.033058 22987 catalog_manager.cc:5484] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 reported cstate change: config changed from index 257 to 258, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3db3750c037e4393927784c55f66d349" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:08.108456 23267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df89654f87ed43758ba68a6a5582b059"
dest_uuid: "3db3750c037e4393927784c55f66d349"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59636
I20240502 20:46:08.108625 23267 raft_consensus.cc:595] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:08.112025 26404 raft_consensus.cc:1050] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: attempting to promote NON_VOTER 1469234e70794be0b1c69b7e7ceb219c to VOTER
I20240502 20:46:08.112418 26404 consensus_queue.cc:237] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.117125 26405 raft_consensus.cc:1050] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68: attempting to promote NON_VOTER 1469234e70794be0b1c69b7e7ceb219c to VOTER
I20240502 20:46:08.117595 26405 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 199, Committed index: 199, Last appended: 3.199, Last appended by leader: 120, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.118996 26093 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c [term 3 LEARNER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 3 index: 199. Preceding OpId from leader: term: 3 index: 200. (index mismatch)
I20240502 20:46:08.119006 24690 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 3 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 3 index: 199. Preceding OpId from leader: term: 3 index: 200. (index mismatch)
I20240502 20:46:08.119417 26261 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 199, Time since last communication: 0.000s
I20240502 20:46:08.119967 26261 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 199, Time since last communication: 0.000s
I20240502 20:46:08.120934 24689 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240502 20:46:08.121455 26094 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 1 LEARNER]: Refusing update from remote peer ee6fe9bcf23a450cb5432a8205660598: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240502 20:46:08.121455 26261 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 3 LEADER]: Committing config change with OpId 3.200: config changed from index 199 to 200, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.121925 25905 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240502 20:46:08.123654 25905 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240502 20:46:08.124894 22980 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 199 to 200, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.126035 26402 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Committing config change with OpId 1.219: config changed from index 218 to 219, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.126052 24689 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 3 FOLLOWER]: Committing config change with OpId 3.200: config changed from index 199 to 200, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.127441 26093 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 218 to 219, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.128242 24689 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 218 to 219, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.130498 22979 catalog_manager.cc:5484] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c reported cstate change: config changed from index 218 to 219, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee6fe9bcf23a450cb5432a8205660598" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.130753 26093 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c [term 3 FOLLOWER]: Committing config change with OpId 3.200: config changed from index 199 to 200, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.134608 24824 consensus_queue.cc:237] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 3.200, Last appended by leader: 120, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.135779 26093 raft_consensus.cc:1261] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c [term 3 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 3 index: 200. Preceding OpId from leader: term: 3 index: 201. (index mismatch)
I20240502 20:46:08.136226 26406 consensus_queue.cc:1035] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20240502 20:46:08.137858 26261 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 [term 3 LEADER]: Committing config change with OpId 3.201: config changed from index 200 to 201, VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) evicted. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.138350 26093 raft_consensus.cc:2932] T be76bba97022462a86ed4a4cc2a2dd64 P 1469234e70794be0b1c69b7e7ceb219c [term 3 FOLLOWER]: Committing config change with OpId 3.201: config changed from index 200 to 201, VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) evicted. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.139273 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet be76bba97022462a86ed4a4cc2a2dd64 with cas_config_opid_index 200: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:08.140399 22980 catalog_manager.cc:5484] T be76bba97022462a86ed4a4cc2a2dd64 P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 200 to 201, VOTER c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134) evicted. New cstate: current_term: 3 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:08.147142 24670 tablet_service.cc:1514] Processing DeleteTablet for tablet be76bba97022462a86ed4a4cc2a2dd64 with delete_type TABLET_DATA_TOMBSTONED (TS c9af0b94bfe2492bb34c7255879a7328 not found in new config with opid_index 201) from {username='slave'} at 127.0.0.1:58084
I20240502 20:46:08.147913 26473 tablet_replica.cc:330] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: stopping tablet replica
I20240502 20:46:08.148242 26473 raft_consensus.cc:2229] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.148480 26473 raft_consensus.cc:2258] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.150461 26473 ts_tablet_manager.cc:1875] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:08.152464 26473 ts_tablet_manager.cc:1888] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.200
I20240502 20:46:08.152561 26473 log.cc:1196] T be76bba97022462a86ed4a4cc2a2dd64 P c9af0b94bfe2492bb34c7255879a7328: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal/wals/be76bba97022462a86ed4a4cc2a2dd64
I20240502 20:46:08.153272 22974 catalog_manager.cc:4830] TS c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861): tablet be76bba97022462a86ed4a4cc2a2dd64 (table rebalance_test_table_3 [id=e6d084e24e8342d892514572e869edd1]) successfully deleted
I20240502 20:46:08.339344 23404 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a5fdd7816e345038ed9b6946084a67e"
dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54912
I20240502 20:46:08.339536 23404 raft_consensus.cc:595] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:08.349822 26261 raft_consensus.cc:1050] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: attempting to promote NON_VOTER 1469234e70794be0b1c69b7e7ceb219c to VOTER
I20240502 20:46:08.350255 26261 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 2.214, Last appended by leader: 47, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.351974 24555 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20240502 20:46:08.352206 26093 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [term 2 LEARNER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20240502 20:46:08.352409 26261 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20240502 20:46:08.352821 26406 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20240502 20:46:08.353267 23267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df89654f87ed43758ba68a6a5582b059"
dest_uuid: "3db3750c037e4393927784c55f66d349"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59654
I20240502 20:46:08.353442 23267 raft_consensus.cc:595] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:08.355309 26406 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.215: config changed from index 214 to 215, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.358078 22980 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 reported cstate change: config changed from index 214 to 215, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.360123 24555 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 214 to 215, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.362203 26093 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 214 to 215, 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.371438 24824 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30"
dest_uuid: "09f2f86617004c96b1e186759506cc68"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59428
I20240502 20:46:08.371654 24824 raft_consensus.cc:595] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:46:08.385026 23404 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a5fdd7816e345038ed9b6946084a67e"
dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54932
I20240502 20:46:08.385504 23404 raft_consensus.cc:595] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:08.400390 23267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df89654f87ed43758ba68a6a5582b059"
dest_uuid: "3db3750c037e4393927784c55f66d349"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59662
I20240502 20:46:08.400600 23267 raft_consensus.cc:595] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:08.504906 26404 raft_consensus.cc:979] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: : Instructing follower 1469234e70794be0b1c69b7e7ceb219c to start an election
I20240502 20:46:08.505117 25905 raft_consensus.cc:1067] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Signalling peer 1469234e70794be0b1c69b7e7ceb219c to start an election
I20240502 20:46:08.505744 26093 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a5fdd7816e345038ed9b6946084a67e"
dest_uuid: "1469234e70794be0b1c69b7e7ceb219c"
 from {username='slave'} at 127.19.66.131:44431
I20240502 20:46:08.505955 26093 raft_consensus.cc:482] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:08.506074 26093 raft_consensus.cc:3037] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:08.507048 26093 raft_consensus.cc:504] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.508153 26093 leader_election.cc:288] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [CANDIDATE]: Term 2 election: Requested vote from peers ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261), c9af0b94bfe2492bb34c7255879a7328 (127.19.66.134:43861)
I20240502 20:46:08.508627 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5fdd7816e345038ed9b6946084a67e" candidate_uuid: "1469234e70794be0b1c69b7e7ceb219c" candidate_term: 2 candidate_status { last_received { term: 1 index: 219 } } ignore_live_leader: true dest_uuid: "ee6fe9bcf23a450cb5432a8205660598"
I20240502 20:46:08.508788 23404 raft_consensus.cc:3032] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:08.508852 23404 raft_consensus.cc:729] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee6fe9bcf23a450cb5432a8205660598, State: Running, Role: LEADER
I20240502 20:46:08.509042 23404 consensus_queue.cc:260] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 1.219, Last appended by leader: 219, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.509299 23404 raft_consensus.cc:3037] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:08.510389 23404 raft_consensus.cc:2454] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1469234e70794be0b1c69b7e7ceb219c in term 2.
I20240502 20:46:08.510869 26025 leader_election.cc:302] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1469234e70794be0b1c69b7e7ceb219c, ee6fe9bcf23a450cb5432a8205660598; no voters: 
I20240502 20:46:08.512390 26358 raft_consensus.cc:2781] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:08.512662 26358 raft_consensus.cc:686] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 2 LEADER]: Becoming Leader. State: Replica: 1469234e70794be0b1c69b7e7ceb219c, State: Running, Role: LEADER
I20240502 20:46:08.513017 26358 consensus_queue.cc:237] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 1.219, Last appended by leader: 219, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.513293 24689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5fdd7816e345038ed9b6946084a67e" candidate_uuid: "1469234e70794be0b1c69b7e7ceb219c" candidate_term: 2 candidate_status { last_received { term: 1 index: 219 } } ignore_live_leader: true dest_uuid: "c9af0b94bfe2492bb34c7255879a7328"
I20240502 20:46:08.513413 24689 raft_consensus.cc:3037] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:08.514494 24689 raft_consensus.cc:2454] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1469234e70794be0b1c69b7e7ceb219c in term 2.
I20240502 20:46:08.516322 22987 catalog_manager.cc:5484] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c reported cstate change: term changed from 1 to 2, leader changed from ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) to 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136). New cstate: current_term: 2 leader_uuid: "1469234e70794be0b1c69b7e7ceb219c" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:08.521917 26429 raft_consensus.cc:979] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: : Instructing follower 09f2f86617004c96b1e186759506cc68 to start an election
I20240502 20:46:08.522078 26429 raft_consensus.cc:1067] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Signalling peer 09f2f86617004c96b1e186759506cc68 to start an election
I20240502 20:46:08.522814 24824 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df89654f87ed43758ba68a6a5582b059"
dest_uuid: "09f2f86617004c96b1e186759506cc68"
 from {username='slave'} at 127.19.66.130:38299
I20240502 20:46:08.522996 24824 raft_consensus.cc:482] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:08.523072 24824 raft_consensus.cc:3037] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:08.526345 24824 raft_consensus.cc:504] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.527212 24824 leader_election.cc:288] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 2 election: Requested vote from peers 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415), 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136:37607)
I20240502 20:46:08.527904 26093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df89654f87ed43758ba68a6a5582b059" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 2 candidate_status { last_received { term: 1 index: 258 } } ignore_live_leader: true dest_uuid: "1469234e70794be0b1c69b7e7ceb219c"
I20240502 20:46:08.527956 23267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df89654f87ed43758ba68a6a5582b059" candidate_uuid: "09f2f86617004c96b1e186759506cc68" candidate_term: 2 candidate_status { last_received { term: 1 index: 258 } } ignore_live_leader: true dest_uuid: "3db3750c037e4393927784c55f66d349"
I20240502 20:46:08.528133 23267 raft_consensus.cc:3032] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:08.528193 23267 raft_consensus.cc:729] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3db3750c037e4393927784c55f66d349, State: Running, Role: LEADER
I20240502 20:46:08.528182 26093 raft_consensus.cc:3037] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:08.528420 23267 consensus_queue.cc:260] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 1.258, Last appended by leader: 258, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.528604 23267 raft_consensus.cc:3037] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:08.529299 26093 raft_consensus.cc:2454] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 2.
I20240502 20:46:08.529779 23267 raft_consensus.cc:2454] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f2f86617004c96b1e186759506cc68 in term 2.
I20240502 20:46:08.529845 24754 leader_election.cc:302] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f2f86617004c96b1e186759506cc68, 1469234e70794be0b1c69b7e7ceb219c; no voters: 
I20240502 20:46:08.530157 26406 raft_consensus.cc:2781] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:08.530332 26406 raft_consensus.cc:686] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Becoming Leader. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Running, Role: LEADER
I20240502 20:46:08.530544 26406 consensus_queue.cc:237] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 1.258, Last appended by leader: 258, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.532630 22987 catalog_manager.cc:5484] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 reported cstate change: term changed from 1 to 2, leader changed from 3db3750c037e4393927784c55f66d349 (127.19.66.130) to 09f2f86617004c96b1e186759506cc68 (127.19.66.135). New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.616582 24824 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30"
dest_uuid: "09f2f86617004c96b1e186759506cc68"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59444
I20240502 20:46:08.616853 24824 raft_consensus.cc:595] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Received request to transfer leadership
I20240502 20:46:08.764180 26406 raft_consensus.cc:979] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: : Instructing follower b30503059718458bb016185550f5e0fb to start an election
I20240502 20:46:08.764369 26406 raft_consensus.cc:1067] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Signalling peer b30503059718458bb016185550f5e0fb to start an election
I20240502 20:46:08.765048 24555 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30"
dest_uuid: "b30503059718458bb016185550f5e0fb"
 from {username='slave'} at 127.19.66.135:55733
I20240502 20:46:08.765234 24555 raft_consensus.cc:482] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:08.765286 24555 raft_consensus.cc:3037] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:08.766284 24555 raft_consensus.cc:504] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.766819 24555 leader_election.cc:288] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [CANDIDATE]: Term 3 election: Requested vote from peers 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917), 1469234e70794be0b1c69b7e7ceb219c (127.19.66.136:37607)
I20240502 20:46:08.767257 24824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30" candidate_uuid: "b30503059718458bb016185550f5e0fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 215 } } ignore_live_leader: true dest_uuid: "09f2f86617004c96b1e186759506cc68"
I20240502 20:46:08.767464 24824 raft_consensus.cc:3032] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 20:46:08.767513 24824 raft_consensus.cc:729] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09f2f86617004c96b1e186759506cc68, State: Running, Role: LEADER
I20240502 20:46:08.767699 24824 consensus_queue.cc:260] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 2.215, Last appended by leader: 215, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.767920 24824 raft_consensus.cc:3037] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:08.769685 24824 raft_consensus.cc:2454] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30503059718458bb016185550f5e0fb in term 3.
I20240502 20:46:08.770228 24488 leader_election.cc:302] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f2f86617004c96b1e186759506cc68, b30503059718458bb016185550f5e0fb; no voters: 
I20240502 20:46:08.770485 26380 raft_consensus.cc:2781] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:46:08.770692 26380 raft_consensus.cc:686] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 3 LEADER]: Becoming Leader. State: Replica: b30503059718458bb016185550f5e0fb, State: Running, Role: LEADER
I20240502 20:46:08.770884 26380 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 2.215, Last appended by leader: 215, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.771793 26093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6c6fc06ec341e28a4b9f231d78ac30" candidate_uuid: "b30503059718458bb016185550f5e0fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 215 } } ignore_live_leader: true dest_uuid: "1469234e70794be0b1c69b7e7ceb219c"
I20240502 20:46:08.771991 26093 raft_consensus.cc:3037] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:08.773049 26093 raft_consensus.cc:2454] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30503059718458bb016185550f5e0fb in term 3.
I20240502 20:46:08.773135 22987 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb reported cstate change: term changed from 2 to 3, leader changed from 09f2f86617004c96b1e186759506cc68 (127.19.66.135) to b30503059718458bb016185550f5e0fb (127.19.66.133). New cstate: current_term: 3 leader_uuid: "b30503059718458bb016185550f5e0fb" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:08.954619 24689 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Refusing update from remote peer 1469234e70794be0b1c69b7e7ceb219c: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240502 20:46:08.955577 26358 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240502 20:46:08.959887 23404 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Refusing update from remote peer 1469234e70794be0b1c69b7e7ceb219c: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240502 20:46:08.960717 26358 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6fe9bcf23a450cb5432a8205660598" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 39261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240502 20:46:08.966078 26093 consensus_queue.cc:237] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 2.220, Last appended by leader: 219, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:08.967486 24689 raft_consensus.cc:1261] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Refusing update from remote peer 1469234e70794be0b1c69b7e7ceb219c: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20240502 20:46:08.968008 26547 consensus_queue.cc:1035] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20240502 20:46:08.970288 26358 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c [term 2 LEADER]: Committing config change with OpId 2.221: config changed from index 219 to 221, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.970868 24689 raft_consensus.cc:2932] T 5a5fdd7816e345038ed9b6946084a67e P c9af0b94bfe2492bb34c7255879a7328 [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 219 to 221, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:08.972385 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a5fdd7816e345038ed9b6946084a67e with cas_config_opid_index 219: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:08.973737 22987 catalog_manager.cc:5484] T 5a5fdd7816e345038ed9b6946084a67e P 1469234e70794be0b1c69b7e7ceb219c reported cstate change: config changed from index 219 to 221, VOTER ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131) evicted. New cstate: current_term: 2 leader_uuid: "1469234e70794be0b1c69b7e7ceb219c" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "c9af0b94bfe2492bb34c7255879a7328" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 43861 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:08.977485 23382 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5fdd7816e345038ed9b6946084a67e with delete_type TABLET_DATA_TOMBSTONED (TS ee6fe9bcf23a450cb5432a8205660598 not found in new config with opid_index 221) from {username='slave'} at 127.0.0.1:39856
I20240502 20:46:08.977926 26595 tablet_replica.cc:330] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: stopping tablet replica
I20240502 20:46:08.978268 26595 raft_consensus.cc:2229] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:08.978483 26595 raft_consensus.cc:2258] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:08.980195 26595 ts_tablet_manager.cc:1875] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:08.982308 26595 ts_tablet_manager.cc:1888] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.220
I20240502 20:46:08.982399 26595 log.cc:1196] T 5a5fdd7816e345038ed9b6946084a67e P ee6fe9bcf23a450cb5432a8205660598: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/5a5fdd7816e345038ed9b6946084a67e
I20240502 20:46:08.983036 22971 catalog_manager.cc:4830] TS ee6fe9bcf23a450cb5432a8205660598 (127.19.66.131:39261): tablet 5a5fdd7816e345038ed9b6946084a67e (table rebalance_test_table_4 [id=ceec311346a6484f9413d765ea4f9f32]) successfully deleted
I20240502 20:46:09.097676 26093 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 2 index: 259. (index mismatch)
I20240502 20:46:09.098192 26406 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240502 20:46:09.101847 23267 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 2 index: 259. (index mismatch)
I20240502 20:46:09.102547 26261 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db3750c037e4393927784c55f66d349" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 35415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240502 20:46:09.104779 24824 consensus_queue.cc:237] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 2.259, Last appended by leader: 258, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:09.106426 26093 raft_consensus.cc:1261] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [term 2 FOLLOWER]: Refusing update from remote peer 09f2f86617004c96b1e186759506cc68: Log matching property violated. Preceding OpId in replica: term: 2 index: 259. Preceding OpId from leader: term: 2 index: 260. (index mismatch)
I20240502 20:46:09.107246 26261 consensus_queue.cc:1035] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20240502 20:46:09.109378 26406 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 09f2f86617004c96b1e186759506cc68 [term 2 LEADER]: Committing config change with OpId 2.260: config changed from index 258 to 260, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:09.109898 26093 raft_consensus.cc:2932] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c [term 2 FOLLOWER]: Committing config change with OpId 2.260: config changed from index 258 to 260, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:09.111126 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df89654f87ed43758ba68a6a5582b059 with cas_config_opid_index 258: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:09.112619 22987 catalog_manager.cc:5484] T df89654f87ed43758ba68a6a5582b059 P 1469234e70794be0b1c69b7e7ceb219c reported cstate change: config changed from index 258 to 260, VOTER 3db3750c037e4393927784c55f66d349 (127.19.66.130) evicted. New cstate: current_term: 2 leader_uuid: "09f2f86617004c96b1e186759506cc68" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:09.116559 23247 tablet_service.cc:1514] Processing DeleteTablet for tablet df89654f87ed43758ba68a6a5582b059 with delete_type TABLET_DATA_TOMBSTONED (TS 3db3750c037e4393927784c55f66d349 not found in new config with opid_index 260) from {username='slave'} at 127.0.0.1:44302
I20240502 20:46:09.123656 26628 tablet_replica.cc:330] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: stopping tablet replica
I20240502 20:46:09.123885 26628 raft_consensus.cc:2229] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.124043 26628 raft_consensus.cc:2258] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.126304 26628 ts_tablet_manager.cc:1875] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:09.129446 26628 ts_tablet_manager.cc:1888] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.259
I20240502 20:46:09.129557 26628 log.cc:1196] T df89654f87ed43758ba68a6a5582b059 P 3db3750c037e4393927784c55f66d349: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/df89654f87ed43758ba68a6a5582b059
I20240502 20:46:09.130331 22972 catalog_manager.cc:4830] TS 3db3750c037e4393927784c55f66d349 (127.19.66.130:35415): tablet df89654f87ed43758ba68a6a5582b059 (table rebalance_test_table_2 [id=16f6b8a52dc94e42a53292a33382ae40]) successfully deleted
I20240502 20:46:09.159992 24823 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 3 FOLLOWER]: Refusing update from remote peer b30503059718458bb016185550f5e0fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 215. Preceding OpId from leader: term: 3 index: 216. (index mismatch)
I20240502 20:46:09.161036 26380 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f2f86617004c96b1e186759506cc68" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 35917 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20240502 20:46:09.164678 26093 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [term 3 FOLLOWER]: Refusing update from remote peer b30503059718458bb016185550f5e0fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 215. Preceding OpId from leader: term: 3 index: 216. (index mismatch)
I20240502 20:46:09.165292 26571 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20240502 20:46:09.639910 24555 consensus_queue.cc:237] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 3.216, Last appended by leader: 215, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } }
I20240502 20:46:09.641372 26093 raft_consensus.cc:1261] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [term 3 FOLLOWER]: Refusing update from remote peer b30503059718458bb016185550f5e0fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 216. Preceding OpId from leader: term: 3 index: 217. (index mismatch)
I20240502 20:46:09.642056 26630 consensus_queue.cc:1035] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20240502 20:46:09.644263 26571 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb [term 3 LEADER]: Committing config change with OpId 3.217: config changed from index 215 to 217, VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:09.644729 26093 raft_consensus.cc:2932] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 1469234e70794be0b1c69b7e7ceb219c [term 3 FOLLOWER]: Committing config change with OpId 3.217: config changed from index 215 to 217, VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } } }
I20240502 20:46:09.647675 22971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 with cas_config_opid_index 215: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:09.648535 22980 catalog_manager.cc:5484] T 0f6c6fc06ec341e28a4b9f231d78ac30 P b30503059718458bb016185550f5e0fb reported cstate change: config changed from index 215 to 217, VOTER 09f2f86617004c96b1e186759506cc68 (127.19.66.135) evicted. New cstate: current_term: 3 leader_uuid: "b30503059718458bb016185550f5e0fb" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "b30503059718458bb016185550f5e0fb" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45729 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1469234e70794be0b1c69b7e7ceb219c" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 37607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:09.653273 24790 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6c6fc06ec341e28a4b9f231d78ac30 with delete_type TABLET_DATA_TOMBSTONED (TS 09f2f86617004c96b1e186759506cc68 not found in new config with opid_index 217) from {username='slave'} at 127.0.0.1:44132
I20240502 20:46:09.654089 26654 tablet_replica.cc:330] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: stopping tablet replica
I20240502 20:46:09.654518 26654 raft_consensus.cc:2229] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:09.654729 26654 raft_consensus.cc:2258] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:09.657088 26654 ts_tablet_manager.cc:1875] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:09.659735 26654 ts_tablet_manager.cc:1888] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.216
I20240502 20:46:09.659845 26654 log.cc:1196] T 0f6c6fc06ec341e28a4b9f231d78ac30 P 09f2f86617004c96b1e186759506cc68: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal/wals/0f6c6fc06ec341e28a4b9f231d78ac30
I20240502 20:46:09.660604 22971 catalog_manager.cc:4830] TS 09f2f86617004c96b1e186759506cc68 (127.19.66.135:35917): tablet 0f6c6fc06ec341e28a4b9f231d78ac30 (table rebalance_test_table_1 [id=08ece7d56fd64ccaabd549fe6b552b96]) successfully deleted
I20240502 20:46:09.832360 26074 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:09.833454 24536 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:09.835918 24790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:09.843369 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:09.855417 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:09.856245 23247 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:09.867509 23382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:09.867506 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:09.987486 26074 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:09.987478 24536 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:09.987782 24790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:09.987790 23247 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:10.001610 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:10.002548 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:10.010437 23382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:10.014583 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:10.145620 23113 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:10.149322 24670 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:10.150476 24790 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:10.157899 23247 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 20:46:10.158885 26074 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:10.164940 24536 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:10.184607 23382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 20:46:10.191303 24402 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 2aa045eae84748a59afc512c822816b6 | 127.19.66.190:46265 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                 |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                 | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                 | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                    | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                 | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                 | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                 | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                   | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                  | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.19.66.148:41037 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 09f2f86617004c96b1e186759506cc68 | 127.19.66.135:35917 | HEALTHY | <none>   |       4        |       0
 140dd1ea33be47e99530c773c5439d56 | 127.19.66.129:45439 | HEALTHY | <none>   |       3        |       0
 1469234e70794be0b1c69b7e7ceb219c | 127.19.66.136:37607 | HEALTHY | <none>   |       1        |       0
 3db3750c037e4393927784c55f66d349 | 127.19.66.130:35415 | HEALTHY | <none>   |       4        |       0
 94ac80c7dd7f4767bb357b89c547d8b2 | 127.19.66.132:45001 | HEALTHY | <none>   |       0        |       0
 b30503059718458bb016185550f5e0fb | 127.19.66.133:45729 | HEALTHY | <none>   |       2        |       0
 c9af0b94bfe2492bb34c7255879a7328 | 127.19.66.134:43861 | HEALTHY | <none>   |       0        |       0
 ee6fe9bcf23a450cb5432a8205660598 | 127.19.66.131:39261 | HEALTHY | <none>   |       2        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       8

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                               |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                              | experimental          | all 8 server(s) checked
 local_ip_for_outbound_sockets            | 127.19.66.129                                                                                                                                                    | experimental          | 127.19.66.129:45439
 local_ip_for_outbound_sockets            | 127.19.66.130                                                                                                                                                    | experimental          | 127.19.66.130:35415
 local_ip_for_outbound_sockets            | 127.19.66.131                                                                                                                                                    | experimental          | 127.19.66.131:39261
 local_ip_for_outbound_sockets            | 127.19.66.132                                                                                                                                                    | experimental          | 127.19.66.132:45001
 local_ip_for_outbound_sockets            | 127.19.66.133                                                                                                                                                    | experimental          | 127.19.66.133:45729
 local_ip_for_outbound_sockets            | 127.19.66.134                                                                                                                                                    | experimental          | 127.19.66.134:43861
 local_ip_for_outbound_sockets            | 127.19.66.135                                                                                                                                                    | experimental          | 127.19.66.135:35917
 local_ip_for_outbound_sockets            | 127.19.66.136                                                                                                                                                    | experimental          | 127.19.66.136:37607
 never_fsync                              | true                                                                                                                                                             | unsafe,advanced       | all 8 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                | unsafe,hidden         | all 8 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                             | experimental,advanced | all 8 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                             | unsafe                | all 8 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                               | hidden                | all 8 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.19.66.129:45439
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.19.66.130:35415
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.19.66.131:39261
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.19.66.132:45001
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.19.66.133:45729
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.19.66.134:43861
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.19.66.135:35917
 server_dump_info_path                    | /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden                | 127.19.66.136:37607

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.19.66.148:41037 | all 8 server(s) checked
 time_source         | builtin             | all 8 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 9 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 4
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 8
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 20:46:10.221431 19722 log_verifier.cc:126] Checking tablet 0f6c6fc06ec341e28a4b9f231d78ac30
I20240502 20:46:10.269716 19722 log_verifier.cc:177] Verified matching terms for 217 ops in tablet 0f6c6fc06ec341e28a4b9f231d78ac30
I20240502 20:46:10.269836 19722 log_verifier.cc:126] Checking tablet 1e226ee7a67b46f687971e7de8099217
I20240502 20:46:10.316314 19722 log_verifier.cc:177] Verified matching terms for 196 ops in tablet 1e226ee7a67b46f687971e7de8099217
I20240502 20:46:10.316457 19722 log_verifier.cc:126] Checking tablet 21b3fcb144894a818b2f649a4473a7ee
I20240502 20:46:10.365976 19722 log_verifier.cc:177] Verified matching terms for 215 ops in tablet 21b3fcb144894a818b2f649a4473a7ee
I20240502 20:46:10.366144 19722 log_verifier.cc:126] Checking tablet 37ab508cf18c46d8ad1775086354f379
I20240502 20:46:10.417591 19722 log_verifier.cc:177] Verified matching terms for 259 ops in tablet 37ab508cf18c46d8ad1775086354f379
I20240502 20:46:10.417757 19722 log_verifier.cc:126] Checking tablet 4b80c0d7e5de4ad58511694ec615dfef
I20240502 20:46:10.469702 19722 log_verifier.cc:177] Verified matching terms for 260 ops in tablet 4b80c0d7e5de4ad58511694ec615dfef
I20240502 20:46:10.469856 19722 log_verifier.cc:126] Checking tablet 4d54cbc40bcc4fa481a04d100ed3c927
I20240502 20:46:10.481262 19722 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 4d54cbc40bcc4fa481a04d100ed3c927
I20240502 20:46:10.481340 19722 log_verifier.cc:126] Checking tablet 5a5fdd7816e345038ed9b6946084a67e
I20240502 20:46:10.533520 19722 log_verifier.cc:177] Verified matching terms for 221 ops in tablet 5a5fdd7816e345038ed9b6946084a67e
I20240502 20:46:10.533695 19722 log_verifier.cc:126] Checking tablet 711786372ea5495abebec7086f958fef
I20240502 20:46:10.583132 19722 log_verifier.cc:177] Verified matching terms for 231 ops in tablet 711786372ea5495abebec7086f958fef
I20240502 20:46:10.583274 19722 log_verifier.cc:126] Checking tablet 798923743c9d444f917b747309d2285c
I20240502 20:46:10.638489 19722 log_verifier.cc:177] Verified matching terms for 252 ops in tablet 798923743c9d444f917b747309d2285c
I20240502 20:46:10.638662 19722 log_verifier.cc:126] Checking tablet 96ee320aaf2c40a9885f934c5c653c78
I20240502 20:46:10.696776 19722 log_verifier.cc:177] Verified matching terms for 270 ops in tablet 96ee320aaf2c40a9885f934c5c653c78
I20240502 20:46:10.696928 19722 log_verifier.cc:126] Checking tablet 98e34e4543d24b46963049828877d3d5
I20240502 20:46:10.739109 19722 log_verifier.cc:177] Verified matching terms for 189 ops in tablet 98e34e4543d24b46963049828877d3d5
I20240502 20:46:10.739233 19722 log_verifier.cc:126] Checking tablet b0f06fbb9f58417793a6d176b1f1078f
I20240502 20:46:10.786984 19722 log_verifier.cc:177] Verified matching terms for 226 ops in tablet b0f06fbb9f58417793a6d176b1f1078f
I20240502 20:46:10.787178 19722 log_verifier.cc:126] Checking tablet be76bba97022462a86ed4a4cc2a2dd64
I20240502 20:46:10.834735 19722 log_verifier.cc:177] Verified matching terms for 201 ops in tablet be76bba97022462a86ed4a4cc2a2dd64
I20240502 20:46:10.834864 19722 log_verifier.cc:126] Checking tablet ddfa7e208cb1404ba25ebc77a75f7c1a
I20240502 20:46:10.888600 19722 log_verifier.cc:177] Verified matching terms for 240 ops in tablet ddfa7e208cb1404ba25ebc77a75f7c1a
I20240502 20:46:10.888741 19722 log_verifier.cc:126] Checking tablet df89654f87ed43758ba68a6a5582b059
I20240502 20:46:10.942209 19722 log_verifier.cc:177] Verified matching terms for 260 ops in tablet df89654f87ed43758ba68a6a5582b059
I20240502 20:46:10.942339 19722 log_verifier.cc:126] Checking tablet fb9d185d01e44dd49580eeb4ab979346
I20240502 20:46:10.986232 19722 log_verifier.cc:177] Verified matching terms for 215 ops in tablet fb9d185d01e44dd49580eeb4ab979346
I20240502 20:46:10.987931 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 23048
I20240502 20:46:10.998613 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 23183
I20240502 20:46:11.007871 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 23318
I20240502 20:46:11.016302 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 24336
I20240502 20:46:11.023669 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 24472
I20240502 20:46:11.030778 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 24607
I20240502 20:46:11.038645 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 24739
I20240502 20:46:11.046581 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 26009
I20240502 20:46:11.053509 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 22956
2024-05-02T20:46:11Z chronyd exiting
[       OK ] RebalanceParamTest.Rebalance/3 (25550 ms)
[----------] 1 test from RebalanceParamTest (25551 ms total)

[----------] 1 test from ConcurrentRebalancersTest
[ RUN      ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240502 20:46:11.117756 19722 test_util.cc:268] Using random seed: -1317431630
I20240502 20:46:11.118646 19722 ts_itest-base.cc:112] Starting cluster with:
I20240502 20:46:11.118698 19722 ts_itest-base.cc:113] --------------
I20240502 20:46:11.118711 19722 ts_itest-base.cc:114] 8 tablet servers
I20240502 20:46:11.118724 19722 ts_itest-base.cc:115] 3 replicas per TS
I20240502 20:46:11.118736 19722 ts_itest-base.cc:116] --------------
2024-05-02T20:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T20:46:11Z Disabled control of system clock
I20240502 20:46:11.131361 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.190:43237
--webserver_interface=127.19.66.190
--webserver_port=0
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.190:43237
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 20:46:11.256377 26777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:11.256691 26777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:11.256767 26777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:11.260089 26777 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:11.260205 26777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 20:46:11.260249 26777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:11.260279 26777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 20:46:11.260320 26777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 20:46:11.264446 26777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.19.66.190:43237
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.190:43237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.19.66.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.26777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:11.265630 26777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:11.266942 26777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:11.276280 26784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:11.277156 26783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:11.277529 26786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:11.278620 26777 server_base.cc:1034] running on GCE node
I20240502 20:46:11.279568 26777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:11.281355 26777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:11.282619 26777 hybrid_clock.cc:648] HybridClock initialized: now 1714682771282588 us; error 61 us; skew 500 ppm
I20240502 20:46:11.285336 26777 webserver.cc:460] Webserver started at http://127.19.66.190:35249/ using document root <none> and password file <none>
I20240502 20:46:11.286029 26777 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:11.286100 26777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:11.286301 26777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:11.288226 26777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e0c668105a124f0eb7227390002cc15e"
format_stamp: "Formatted at 2024-05-02 20:46:11 on dist-test-slave-gqqg"
I20240502 20:46:11.288753 26777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e0c668105a124f0eb7227390002cc15e"
format_stamp: "Formatted at 2024-05-02 20:46:11 on dist-test-slave-gqqg"
I20240502 20:46:11.293984 26777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20240502 20:46:11.296825 26792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:11.298038 26777 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240502 20:46:11.298193 26777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e0c668105a124f0eb7227390002cc15e"
format_stamp: "Formatted at 2024-05-02 20:46:11 on dist-test-slave-gqqg"
I20240502 20:46:11.298332 26777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:11.322953 26777 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:11.323843 26777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:11.324038 26777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:11.331756 26777 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.190:43237
I20240502 20:46:11.331801 26847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.190:43237 every 8 connection(s)
I20240502 20:46:11.333194 26777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 20:46:11.336560 26848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:11.339320 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 26777
I20240502 20:46:11.339478 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 20:46:11.343199 26848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e: Bootstrap starting.
I20240502 20:46:11.345814 26848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:11.346773 26848 log.cc:824] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:11.348968 26848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e: No bootstrap required, opened a new log
I20240502 20:46:11.351819 26848 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c668105a124f0eb7227390002cc15e" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43237 } }
I20240502 20:46:11.352032 26848 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:11.352079 26848 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c668105a124f0eb7227390002cc15e, State: Initialized, Role: FOLLOWER
I20240502 20:46:11.352548 26848 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c668105a124f0eb7227390002cc15e" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43237 } }
I20240502 20:46:11.352676 26848 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:46:11.352737 26848 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:46:11.352855 26848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:11.353878 26848 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c668105a124f0eb7227390002cc15e" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43237 } }
I20240502 20:46:11.354247 26848 leader_election.cc:302] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0c668105a124f0eb7227390002cc15e; no voters: 
I20240502 20:46:11.354573 26848 leader_election.cc:288] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 20:46:11.354907 26854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:11.355145 26854 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [term 1 LEADER]: Becoming Leader. State: Replica: e0c668105a124f0eb7227390002cc15e, State: Running, Role: LEADER
I20240502 20:46:11.355435 26854 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c668105a124f0eb7227390002cc15e" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43237 } }
I20240502 20:46:11.355756 26848 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:46:11.357023 26855 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0c668105a124f0eb7227390002cc15e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c668105a124f0eb7227390002cc15e" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43237 } } }
I20240502 20:46:11.357057 26856 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0c668105a124f0eb7227390002cc15e. Latest consensus state: current_term: 1 leader_uuid: "e0c668105a124f0eb7227390002cc15e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c668105a124f0eb7227390002cc15e" member_type: VOTER last_known_addr { host: "127.19.66.190" port: 43237 } } }
I20240502 20:46:11.357205 26855 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:11.357208 26856 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e [sys.catalog]: This master's current role is: LEADER
I20240502 20:46:11.357616 26863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:46:11.361117 26863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:46:11.366577 26863 catalog_manager.cc:1347] Generated new cluster ID: fe1a5dc919dd4f64ab92d108f9f7709b
I20240502 20:46:11.366684 26863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:46:11.385178 26863 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:46:11.386485 26863 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:46:11.395051 26863 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e0c668105a124f0eb7227390002cc15e: Generated new TSK 0
I20240502 20:46:11.396296 26863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:46:11.408970 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.129:0
--local_ip_for_outbound_sockets=127.19.66.129
--webserver_interface=127.19.66.129
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:46:11.546743 26873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:11.547037 26873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:11.547101 26873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:11.550618 26873 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:11.550717 26873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:11.550806 26873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.129
I20240502 20:46:11.555732 26873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.19.66.129
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.26873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.129
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:11.557658 26873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:11.559247 26873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:11.569486 26884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:11.570050 26881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:11.570856 26879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:11.571630 26873 server_base.cc:1034] running on GCE node
I20240502 20:46:11.572190 26873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:11.572925 26873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:11.574242 26873 hybrid_clock.cc:648] HybridClock initialized: now 1714682771574197 us; error 45 us; skew 500 ppm
I20240502 20:46:11.577466 26873 webserver.cc:460] Webserver started at http://127.19.66.129:43103/ using document root <none> and password file <none>
I20240502 20:46:11.578169 26873 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:11.578248 26873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:11.578480 26873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:11.580569 26873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "297de823852d4a8da44791ab03131cd3"
format_stamp: "Formatted at 2024-05-02 20:46:11 on dist-test-slave-gqqg"
I20240502 20:46:11.581211 26873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "297de823852d4a8da44791ab03131cd3"
format_stamp: "Formatted at 2024-05-02 20:46:11 on dist-test-slave-gqqg"
I20240502 20:46:11.585839 26873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 20:46:11.589360 26890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:11.590993 26873 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 20:46:11.591187 26873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "297de823852d4a8da44791ab03131cd3"
format_stamp: "Formatted at 2024-05-02 20:46:11 on dist-test-slave-gqqg"
I20240502 20:46:11.591328 26873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:11.634929 26873 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:11.635648 26873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:11.635830 26873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:11.636425 26873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:11.637532 26873 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:11.637605 26873 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:11.637678 26873 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:11.637704 26873 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:11.647258 26873 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.129:38835
I20240502 20:46:11.647282 27005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.129:38835 every 8 connection(s)
I20240502 20:46:11.648316 26873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 20:46:11.649351 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 26873
I20240502 20:46:11.649477 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 20:46:11.652048 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.130:0
--local_ip_for_outbound_sockets=127.19.66.130
--webserver_interface=127.19.66.130
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:11.659147 27006 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:11.659420 27006 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:11.660079 27006 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:11.661715 26807 ts_manager.cc:194] Registered new tserver with Master: 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:11.662992 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.129:58265
W20240502 20:46:11.775954 27009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:11.776257 27009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:11.776352 27009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:11.779666 27009 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:11.779762 27009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:11.779878 27009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.130
I20240502 20:46:11.784070 27009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.19.66.130
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.27009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.130
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:11.785310 27009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:11.786640 27009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:11.796937 27014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:11.797451 27017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:11.798182 27015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:11.798861 27009 server_base.cc:1034] running on GCE node
I20240502 20:46:11.799391 27009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:11.800042 27009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:11.801208 27009 hybrid_clock.cc:648] HybridClock initialized: now 1714682771801185 us; error 49 us; skew 500 ppm
I20240502 20:46:11.803530 27009 webserver.cc:460] Webserver started at http://127.19.66.130:41521/ using document root <none> and password file <none>
I20240502 20:46:11.804141 27009 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:11.804209 27009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:11.804423 27009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:11.806351 27009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d1f3f8e444364d6faa42d651b5dbb997"
format_stamp: "Formatted at 2024-05-02 20:46:11 on dist-test-slave-gqqg"
I20240502 20:46:11.806890 27009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d1f3f8e444364d6faa42d651b5dbb997"
format_stamp: "Formatted at 2024-05-02 20:46:11 on dist-test-slave-gqqg"
I20240502 20:46:11.811223 27009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 20:46:11.814360 27023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:11.815691 27009 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:46:11.815846 27009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d1f3f8e444364d6faa42d651b5dbb997"
format_stamp: "Formatted at 2024-05-02 20:46:11 on dist-test-slave-gqqg"
I20240502 20:46:11.815977 27009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:11.858947 27009 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:11.859648 27009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:11.859827 27009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:11.860395 27009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:11.861389 27009 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:11.861445 27009 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:11.861510 27009 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:11.861552 27009 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:11.872432 27009 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.130:43307
I20240502 20:46:11.872498 27139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.130:43307 every 8 connection(s)
I20240502 20:46:11.873546 27009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 20:46:11.880540 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 27009
I20240502 20:46:11.880683 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 20:46:11.883755 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.131:0
--local_ip_for_outbound_sockets=127.19.66.131
--webserver_interface=127.19.66.131
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:11.886885 27140 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:11.887193 27140 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:11.887848 27140 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:11.889207 26807 ts_manager.cc:194] Registered new tserver with Master: d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:11.889845 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.130:54041
W20240502 20:46:12.005738 27143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:12.006047 27143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:12.006136 27143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:12.009615 27143 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:12.009717 27143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:12.009833 27143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.131
I20240502 20:46:12.014732 27143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.19.66.131
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.27143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.131
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:12.016249 27143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:12.018124 27143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:12.028779 27148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:12.029429 27151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:12.029909 27149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:12.030804 27143 server_base.cc:1034] running on GCE node
I20240502 20:46:12.031354 27143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:12.032066 27143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:12.033288 27143 hybrid_clock.cc:648] HybridClock initialized: now 1714682772033252 us; error 45 us; skew 500 ppm
I20240502 20:46:12.036237 27143 webserver.cc:460] Webserver started at http://127.19.66.131:37433/ using document root <none> and password file <none>
I20240502 20:46:12.037029 27143 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:12.037173 27143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:12.037400 27143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:12.039394 27143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c107c43e538d48f58625c4320f0aa4b8"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.039988 27143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c107c43e538d48f58625c4320f0aa4b8"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.044090 27143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20240502 20:46:12.047009 27157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.048187 27143 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:12.048357 27143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c107c43e538d48f58625c4320f0aa4b8"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.048475 27143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:12.078311 27143 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:12.079018 27143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:12.079210 27143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:12.079842 27143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:12.080874 27143 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:12.080931 27143 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.080999 27143 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:12.081029 27143 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.090675 27143 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.131:44123
I20240502 20:46:12.090726 27275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.131:44123 every 8 connection(s)
I20240502 20:46:12.091729 27143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 20:46:12.092864 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 27143
I20240502 20:46:12.093011 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 20:46:12.095496 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.132:0
--local_ip_for_outbound_sockets=127.19.66.132
--webserver_interface=127.19.66.132
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:12.103188 27276 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:12.103485 27276 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:12.104168 27276 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:12.105648 26807 ts_manager.cc:194] Registered new tserver with Master: c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:12.106588 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.131:56265
W20240502 20:46:12.238690 27279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:12.238991 27279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:12.239044 27279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:12.243690 27279 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:12.243786 27279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:12.243863 27279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.132
I20240502 20:46:12.248469 27279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.19.66.132
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.27279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.132
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:12.249609 27279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:12.250841 27279 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:12.260988 27288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:12.261677 27284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:12.262610 27285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:12.263212 27279 server_base.cc:1034] running on GCE node
I20240502 20:46:12.263901 27279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:12.264851 27279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:12.266126 27279 hybrid_clock.cc:648] HybridClock initialized: now 1714682772266070 us; error 82 us; skew 500 ppm
I20240502 20:46:12.275360 27279 webserver.cc:460] Webserver started at http://127.19.66.132:41717/ using document root <none> and password file <none>
I20240502 20:46:12.276227 27279 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:12.276310 27279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:12.276556 27279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:12.278868 27279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.279481 27279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.284514 27279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 20:46:12.288666 27294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.290504 27279 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 20:46:12.290762 27279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.290920 27279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:12.319295 27279 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:12.320194 27279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:12.320398 27279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:12.321204 27279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:12.322448 27279 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:12.322506 27279 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.322572 27279 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:12.322599 27279 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.333875 27279 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.132:41007
I20240502 20:46:12.333912 27408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.132:41007 every 8 connection(s)
I20240502 20:46:12.335348 27279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 20:46:12.345058 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 27279
I20240502 20:46:12.345273 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 20:46:12.348018 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.133:0
--local_ip_for_outbound_sockets=127.19.66.133
--webserver_interface=127.19.66.133
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:12.348955 27409 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:12.349259 27409 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:12.349913 27409 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:12.351122 26807 ts_manager.cc:194] Registered new tserver with Master: ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:12.351761 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.132:45497
W20240502 20:46:12.482828 27412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:12.483124 27412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:12.483206 27412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:12.486706 27412 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:12.486810 27412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:12.486923 27412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.133
I20240502 20:46:12.491000 27412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.66.133
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.27412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.133
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:12.492125 27412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:12.493454 27412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:12.504101 27418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:12.504778 27417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:12.505559 27420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:12.506273 27412 server_base.cc:1034] running on GCE node
I20240502 20:46:12.506809 27412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:12.507550 27412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:12.508733 27412 hybrid_clock.cc:648] HybridClock initialized: now 1714682772508710 us; error 41 us; skew 500 ppm
I20240502 20:46:12.511145 27412 webserver.cc:460] Webserver started at http://127.19.66.133:42147/ using document root <none> and password file <none>
I20240502 20:46:12.511826 27412 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:12.511891 27412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:12.512089 27412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:12.514110 27412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1f40a11333e44ea0a34b6bc7618af522"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.514669 27412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1f40a11333e44ea0a34b6bc7618af522"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.520208 27412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 20:46:12.524312 27426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.526124 27412 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 20:46:12.526302 27412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1f40a11333e44ea0a34b6bc7618af522"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.526470 27412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:12.550316 27412 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:12.551013 27412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:12.551322 27412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:12.551971 27412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:12.553025 27412 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:12.553104 27412 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.553171 27412 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:12.553193 27412 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.562641 27412 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.133:45949
I20240502 20:46:12.562729 27542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.133:45949 every 8 connection(s)
I20240502 20:46:12.563803 27412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:46:12.566480 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 27412
I20240502 20:46:12.566627 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 20:46:12.569169 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.134:0
--local_ip_for_outbound_sockets=127.19.66.134
--webserver_interface=127.19.66.134
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:12.575125 27543 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:12.575578 27543 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:12.576601 27543 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:12.578450 26807 ts_manager.cc:194] Registered new tserver with Master: 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949)
I20240502 20:46:12.579238 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.133:36735
I20240502 20:46:12.665684 27006 heartbeater.cc:503] Master 127.19.66.190:43237 was elected leader, sending a full tablet report...
W20240502 20:46:12.700693 27546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:12.700955 27546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:12.701030 27546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:12.704588 27546 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:12.704733 27546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:12.704840 27546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.134
I20240502 20:46:12.708873 27546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.19.66.134
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.27546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.134
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:12.710034 27546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:12.711529 27546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:12.722151 27551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:12.722723 27552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:12.723423 27554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:12.724148 27546 server_base.cc:1034] running on GCE node
I20240502 20:46:12.724813 27546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:12.725694 27546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:12.726894 27546 hybrid_clock.cc:648] HybridClock initialized: now 1714682772726859 us; error 43 us; skew 500 ppm
I20240502 20:46:12.730309 27546 webserver.cc:460] Webserver started at http://127.19.66.134:46289/ using document root <none> and password file <none>
I20240502 20:46:12.731050 27546 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:12.731117 27546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:12.731324 27546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:12.733304 27546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.733863 27546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.738710 27546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 20:46:12.742693 27560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.744151 27546 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.005s	sys 0.000s
I20240502 20:46:12.744323 27546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.744457 27546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:12.792258 27546 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:12.793058 27546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:12.793316 27546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:12.793978 27546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:12.795190 27546 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:12.795250 27546 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.795316 27546 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:12.795382 27546 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.806339 27546 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.134:46737
I20240502 20:46:12.806378 27677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.134:46737 every 8 connection(s)
I20240502 20:46:12.807446 27546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:46:12.808187 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 27546
I20240502 20:46:12.808369 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 20:46:12.811340 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.135:0
--local_ip_for_outbound_sockets=127.19.66.135
--webserver_interface=127.19.66.135
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:12.819682 27678 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:12.819988 27678 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:12.820708 27678 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:12.822559 26807 ts_manager.cc:194] Registered new tserver with Master: 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:12.823202 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.134:34297
I20240502 20:46:12.892339 27140 heartbeater.cc:503] Master 127.19.66.190:43237 was elected leader, sending a full tablet report...
W20240502 20:46:12.944252 27681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:12.944561 27681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:12.944638 27681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:12.948220 27681 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:12.948340 27681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:12.948464 27681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.135
I20240502 20:46:12.952826 27681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.19.66.135
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.27681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.135
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:12.953963 27681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:12.955225 27681 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:12.965104 27686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:12.965742 27690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:12.966460 27688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:12.967202 27681 server_base.cc:1034] running on GCE node
I20240502 20:46:12.967670 27681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:12.968335 27681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:12.969530 27681 hybrid_clock.cc:648] HybridClock initialized: now 1714682772969513 us; error 29 us; skew 500 ppm
I20240502 20:46:12.972313 27681 webserver.cc:460] Webserver started at http://127.19.66.135:38917/ using document root <none> and password file <none>
I20240502 20:46:12.972968 27681 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:12.973045 27681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:12.973325 27681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:12.975253 27681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "1bc8a142cfc7487bacc573afb0a92b74"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.975813 27681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "1bc8a142cfc7487bacc573afb0a92b74"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.979934 27681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240502 20:46:12.982759 27696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:12.984411 27681 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:46:12.984663 27681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1bc8a142cfc7487bacc573afb0a92b74"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:12.984841 27681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:13.006021 27681 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:13.006786 27681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:13.006982 27681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:13.007622 27681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:13.008661 27681 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:13.008720 27681 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:13.008790 27681 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:13.008816 27681 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:13.017997 27681 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.135:46845
I20240502 20:46:13.018070 27813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.135:46845 every 8 connection(s)
I20240502 20:46:13.019189 27681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:46:13.020741 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 27681
I20240502 20:46:13.020927 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 20:46:13.023440 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.136:0
--local_ip_for_outbound_sockets=127.19.66.136
--webserver_interface=127.19.66.136
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:13.029884 27814 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:13.030164 27814 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:13.031136 27814 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:13.032699 26807 ts_manager.cc:194] Registered new tserver with Master: 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
I20240502 20:46:13.033625 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.135:44775
I20240502 20:46:13.109251 27276 heartbeater.cc:503] Master 127.19.66.190:43237 was elected leader, sending a full tablet report...
W20240502 20:46:13.155191 27817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:13.155484 27817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:13.155561 27817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:13.158919 27817 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:13.159000 27817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:13.159098 27817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.136
I20240502 20:46:13.163022 27817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.19.66.136
--webserver_port=0
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.27817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.136
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:13.164094 27817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:13.165364 27817 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:13.174769 27825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:13.175113 27822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:13.175810 27823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:13.176694 27817 server_base.cc:1034] running on GCE node
I20240502 20:46:13.177269 27817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:13.177942 27817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:13.179126 27817 hybrid_clock.cc:648] HybridClock initialized: now 1714682773179100 us; error 45 us; skew 500 ppm
I20240502 20:46:13.181401 27817 webserver.cc:460] Webserver started at http://127.19.66.136:39859/ using document root <none> and password file <none>
I20240502 20:46:13.182006 27817 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:13.182068 27817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:13.182262 27817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:46:13.184078 27817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "6706d1c0946d42a9bae62d719e1e5c31"
format_stamp: "Formatted at 2024-05-02 20:46:13 on dist-test-slave-gqqg"
I20240502 20:46:13.184619 27817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "6706d1c0946d42a9bae62d719e1e5c31"
format_stamp: "Formatted at 2024-05-02 20:46:13 on dist-test-slave-gqqg"
I20240502 20:46:13.188443 27817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240502 20:46:13.192059 27831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:13.193428 27817 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 20:46:13.193563 27817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6706d1c0946d42a9bae62d719e1e5c31"
format_stamp: "Formatted at 2024-05-02 20:46:13 on dist-test-slave-gqqg"
I20240502 20:46:13.193697 27817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:13.216043 27817 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:13.216749 27817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:13.216917 27817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:13.217547 27817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:13.218554 27817 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:13.218601 27817 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:13.218648 27817 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:13.218680 27817 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:13.227792 27817 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.136:41529
I20240502 20:46:13.227808 27947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.136:41529 every 8 connection(s)
I20240502 20:46:13.228847 27817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:46:13.231343 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 27817
I20240502 20:46:13.231493 19722 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 20:46:13.239004 27948 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:13.239647 27948 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:13.240392 27948 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:13.241788 26807 ts_manager.cc:194] Registered new tserver with Master: 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:13.242434 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.136:44489
I20240502 20:46:13.246250 19722 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 20:46:13.257747 19722 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 20:46:13.257839 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 27412
I20240502 20:46:13.265592 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 27546
I20240502 20:46:13.272059 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 27681
I20240502 20:46:13.278872 19722 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF51BWD/build/debug/bin/kudu with pid 27817
I20240502 20:46:13.354372 27409 heartbeater.cc:503] Master 127.19.66.190:43237 was elected leader, sending a full tablet report...
I20240502 20:46:17.037771 26807 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47424:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:46:17.064296 27342 tablet_service.cc:1467] Processing CreateTablet for tablet d44f271f166b4c288b76689a3e3764d3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.064652 26937 tablet_service.cc:1467] Processing CreateTablet for tablet d44f271f166b4c288b76689a3e3764d3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.064296 26936 tablet_service.cc:1467] Processing CreateTablet for tablet 078a1591d9d74e31a0407aeab76911b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.064301 27341 tablet_service.cc:1467] Processing CreateTablet for tablet 078a1591d9d74e31a0407aeab76911b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.064301 26938 tablet_service.cc:1467] Processing CreateTablet for tablet f2dfc0bbbbca4cc28a97480007343424 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.065759 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44f271f166b4c288b76689a3e3764d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.065775 26937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44f271f166b4c288b76689a3e3764d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.066228 27341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078a1591d9d74e31a0407aeab76911b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.068703 26938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2dfc0bbbbca4cc28a97480007343424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.070116 27207 tablet_service.cc:1467] Processing CreateTablet for tablet f2dfc0bbbbca4cc28a97480007343424 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.070109 27072 tablet_service.cc:1467] Processing CreateTablet for tablet f2dfc0bbbbca4cc28a97480007343424 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.071228 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2dfc0bbbbca4cc28a97480007343424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.071293 27072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2dfc0bbbbca4cc28a97480007343424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.073652 27205 tablet_service.cc:1467] Processing CreateTablet for tablet 078a1591d9d74e31a0407aeab76911b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.073962 27205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078a1591d9d74e31a0407aeab76911b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.074635 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078a1591d9d74e31a0407aeab76911b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.075315 27969 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:17.070119 27071 tablet_service.cc:1467] Processing CreateTablet for tablet d44f271f166b4c288b76689a3e3764d3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.075884 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44f271f166b4c288b76689a3e3764d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.079891 27969 tablet_bootstrap.cc:654] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.081524 27969 log.cc:824] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:17.083741 27969 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:17.083967 27969 ts_tablet_manager.cc:1367] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240502 20:46:17.086514 27973 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:17.087321 27971 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:17.087741 27969 raft_consensus.cc:348] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.088001 27969 raft_consensus.cc:374] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.088342 27969 raft_consensus.cc:729] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.088910 27969 consensus_queue.cc:260] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.089042 27973 tablet_bootstrap.cc:654] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.089763 27971 tablet_bootstrap.cc:654] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.089962 27973 log.cc:824] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:17.089998 27969 ts_tablet_manager.cc:1398] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.003s
I20240502 20:46:17.090421 27969 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:17.090639 27971 log.cc:824] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:17.091730 27969 tablet_bootstrap.cc:654] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.092048 27973 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:17.092268 27973 ts_tablet_manager.cc:1367] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 20:46:17.093137 27969 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:17.093263 27969 ts_tablet_manager.cc:1367] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:17.093822 27969 raft_consensus.cc:348] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:17.093950 27969 raft_consensus.cc:374] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.093983 27969 raft_consensus.cc:729] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.094126 27969 consensus_queue.cc:260] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:17.094549 27969 ts_tablet_manager.cc:1398] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:17.094769 27969 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:17.095862 27969 tablet_bootstrap.cc:654] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.096598 27973 raft_consensus.cc:348] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:17.096868 27973 raft_consensus.cc:374] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.096941 27973 raft_consensus.cc:729] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.097481 27969 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:17.097577 27969 ts_tablet_manager.cc:1367] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:17.097693 27973 consensus_queue.cc:260] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:17.098500 27969 raft_consensus.cc:348] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.098623 27969 raft_consensus.cc:374] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.098665 27969 raft_consensus.cc:729] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.098739 27973 ts_tablet_manager.cc:1398] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240502 20:46:17.098841 27969 consensus_queue.cc:260] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.099112 27969 ts_tablet_manager.cc:1398] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:46:17.099639 27970 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:17.101110 27973 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:17.102260 27973 tablet_bootstrap.cc:654] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.102607 27970 tablet_bootstrap.cc:654] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.103611 27971 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:17.103668 27973 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:17.103682 27970 log.cc:824] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:17.103770 27973 ts_tablet_manager.cc:1367] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:17.104049 27971 ts_tablet_manager.cc:1367] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.017s	user 0.006s	sys 0.000s
I20240502 20:46:17.104416 27973 raft_consensus.cc:348] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.104570 27973 raft_consensus.cc:374] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.104616 27973 raft_consensus.cc:729] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.104780 27973 consensus_queue.cc:260] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.105123 27973 ts_tablet_manager.cc:1398] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:17.105948 27970 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:17.106179 27970 ts_tablet_manager.cc:1367] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.005s
I20240502 20:46:17.108155 27971 raft_consensus.cc:348] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.108404 27971 raft_consensus.cc:374] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.108469 27971 raft_consensus.cc:729] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.108978 27971 consensus_queue.cc:260] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.110360 27970 raft_consensus.cc:348] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:17.110622 27970 raft_consensus.cc:374] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.110698 27970 raft_consensus.cc:729] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.110862 27971 ts_tablet_manager.cc:1398] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20240502 20:46:17.111330 27971 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:17.111243 27970 consensus_queue.cc:260] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:17.112327 27970 ts_tablet_manager.cc:1398] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240502 20:46:17.112632 27971 tablet_bootstrap.cc:654] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.112804 27970 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:17.113926 27970 tablet_bootstrap.cc:654] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.115648 27971 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:17.115823 27971 ts_tablet_manager.cc:1367] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 20:46:17.116535 27971 raft_consensus.cc:348] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.116655 27971 raft_consensus.cc:374] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.116679 27971 raft_consensus.cc:729] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.116807 27971 consensus_queue.cc:260] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.117157 27971 ts_tablet_manager.cc:1398] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.001s
I20240502 20:46:17.117758 27970 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:17.117897 27970 ts_tablet_manager.cc:1367] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:46:17.118697 27970 raft_consensus.cc:348] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.118814 27970 raft_consensus.cc:374] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.118837 27970 raft_consensus.cc:729] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.118971 27970 consensus_queue.cc:260] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.119357 27970 ts_tablet_manager.cc:1398] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
W20240502 20:46:17.129976 27141 tablet.cc:2369] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:46:17.154292 27007 tablet.cc:2369] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:17.160614 27981 raft_consensus.cc:482] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:17.160878 27981 raft_consensus.cc:504] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:17.162446 27981 leader_election.cc:288] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:17.167605 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078a1591d9d74e31a0407aeab76911b4" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:17.167906 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078a1591d9d74e31a0407aeab76911b4" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:17.168036 26960 raft_consensus.cc:2454] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:17.168171 27362 raft_consensus.cc:2454] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:17.168663 27160 leader_election.cc:302] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:17.169027 27981 raft_consensus.cc:2781] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:17.169193 27981 raft_consensus.cc:482] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:17.169231 27981 raft_consensus.cc:3037] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.170485 27981 raft_consensus.cc:504] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:17.171006 27981 leader_election.cc:288] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Requested vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:17.171643 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078a1591d9d74e31a0407aeab76911b4" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:17.171643 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "078a1591d9d74e31a0407aeab76911b4" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:17.171840 27362 raft_consensus.cc:3037] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.171835 26960 raft_consensus.cc:3037] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.173511 27362 raft_consensus.cc:2454] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:17.173795 26960 raft_consensus.cc:2454] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:17.174041 27160 leader_election.cc:302] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:17.174304 27981 raft_consensus.cc:2781] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:17.174604 27981 raft_consensus.cc:686] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Leader. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:17.175083 27981 consensus_queue.cc:237] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:17.179766 26809 catalog_manager.cc:5484] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:17.212992 27978 raft_consensus.cc:482] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:17.213232 27978 raft_consensus.cc:504] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.214529 27978 leader_election.cc:288] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:17.220021 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2dfc0bbbbca4cc28a97480007343424" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:17.220122 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2dfc0bbbbca4cc28a97480007343424" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:17.220361 27228 raft_consensus.cc:2454] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 0.
I20240502 20:46:17.220463 27093 raft_consensus.cc:2454] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 0.
I20240502 20:46:17.220952 26894 leader_election.cc:302] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:17.221449 27978 raft_consensus.cc:2781] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:17.221561 27978 raft_consensus.cc:482] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:17.221614 27978 raft_consensus.cc:3037] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.222698 27978 raft_consensus.cc:504] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.223194 27978 leader_election.cc:288] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 election: Requested vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:17.223748 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2dfc0bbbbca4cc28a97480007343424" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:17.223891 27228 raft_consensus.cc:3037] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.223841 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2dfc0bbbbca4cc28a97480007343424" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:17.224069 27093 raft_consensus.cc:3037] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.225036 27228 raft_consensus.cc:2454] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 1.
I20240502 20:46:17.225432 27093 raft_consensus.cc:2454] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 1.
I20240502 20:46:17.225546 26894 leader_election.cc:302] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:17.225796 27978 raft_consensus.cc:2781] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:17.225941 27978 raft_consensus.cc:686] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Becoming Leader. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Running, Role: LEADER
I20240502 20:46:17.226372 27978 consensus_queue.cc:237] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.230540 26809 catalog_manager.cc:5484] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297de823852d4a8da44791ab03131cd3 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:17.341383 27410 tablet.cc:2369] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 20:46:17.347955 27277 tablet.cc:2369] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 20:46:17.479888 27984 raft_consensus.cc:482] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:17.480058 27984 raft_consensus.cc:504] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.481293 27984 leader_election.cc:288] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:17.485607 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44f271f166b4c288b76689a3e3764d3" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:17.485704 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44f271f166b4c288b76689a3e3764d3" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:17.485790 26960 raft_consensus.cc:2454] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:17.485922 27362 raft_consensus.cc:2454] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:17.486315 27024 leader_election.cc:302] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:17.486754 27984 raft_consensus.cc:2781] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:17.486857 27984 raft_consensus.cc:482] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:17.486920 27984 raft_consensus.cc:3037] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.488826 27984 raft_consensus.cc:504] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.489375 27984 leader_election.cc:288] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Requested vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:17.489936 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44f271f166b4c288b76689a3e3764d3" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:17.489946 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44f271f166b4c288b76689a3e3764d3" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:17.490088 27362 raft_consensus.cc:3037] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.490121 26960 raft_consensus.cc:3037] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.491003 27362 raft_consensus.cc:2454] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:17.491225 26960 raft_consensus.cc:2454] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:17.491479 27026 leader_election.cc:302] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:17.491745 27984 raft_consensus.cc:2781] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:17.492049 27984 raft_consensus.cc:686] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Becoming Leader. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:17.492493 27984 consensus_queue.cc:237] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.496325 26809 catalog_manager.cc:5484] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } }
W20240502 20:46:17.652866 28000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:17.653319 28000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:17.657410 28000 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 20:46:17.705744 28018 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.716225 28024 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.776374 27362 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.778074 27984 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.779588 26959 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.779899 27093 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.780951 27228 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:17.781186 27984 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.781524 27978 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.782119 27978 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:17.806985 28042 mvcc.cc:205] Tried to move back new op lower bound from 7023340657757827072 to 7023340656612814848. Current Snapshot: MvccSnapshot[applied={T|T < 7023340657757827072}]
I20240502 20:46:17.812577 28027 mvcc.cc:205] Tried to move back new op lower bound from 7023340657757827072 to 7023340656612814848. Current Snapshot: MvccSnapshot[applied={T|T < 7023340657757827072}]
I20240502 20:46:17.817945 28031 mvcc.cc:205] Tried to move back new op lower bound from 7023340657783091200 to 7023340656612814848. Current Snapshot: MvccSnapshot[applied={T|T < 7023340657757827072}]

INSERT report
    rows total: 2000
    time total: 136.742 ms
  time per row: 0.0683711 ms
I20240502 20:46:17.862962 26801 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47424:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:46:17.871891 27207 tablet_service.cc:1467] Processing CreateTablet for tablet d1b9ff3c921d474fa5ad106d1c17ab00 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.871891 27341 tablet_service.cc:1467] Processing CreateTablet for tablet d1b9ff3c921d474fa5ad106d1c17ab00 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.872051 27072 tablet_service.cc:1467] Processing CreateTablet for tablet d1b9ff3c921d474fa5ad106d1c17ab00 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.872283 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b9ff3c921d474fa5ad106d1c17ab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.872334 27072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b9ff3c921d474fa5ad106d1c17ab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.873716 27071 tablet_service.cc:1467] Processing CreateTablet for tablet 367543df62e048268908eb0d84c14876 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.874002 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367543df62e048268908eb0d84c14876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.875355 27205 tablet_service.cc:1467] Processing CreateTablet for tablet 367543df62e048268908eb0d84c14876 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.875519 27204 tablet_service.cc:1467] Processing CreateTablet for tablet b466a750cc6b4382947282d8aa6aeb1f (DEFAULT_TABLE table=rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.875644 27205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367543df62e048268908eb0d84c14876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.876518 26937 tablet_service.cc:1467] Processing CreateTablet for tablet b466a750cc6b4382947282d8aa6aeb1f (DEFAULT_TABLE table=rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.876844 26937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b466a750cc6b4382947282d8aa6aeb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.877341 27341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b9ff3c921d474fa5ad106d1c17ab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.877521 27070 tablet_service.cc:1467] Processing CreateTablet for tablet b466a750cc6b4382947282d8aa6aeb1f (DEFAULT_TABLE table=rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.877799 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b466a750cc6b4382947282d8aa6aeb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.879022 28070 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:17.880120 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b466a750cc6b4382947282d8aa6aeb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.880131 28070 tablet_bootstrap.cc:654] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.880209 27342 tablet_service.cc:1467] Processing CreateTablet for tablet 367543df62e048268908eb0d84c14876 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:17.880566 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367543df62e048268908eb0d84c14876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:17.881183 28071 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:17.882153 28071 tablet_bootstrap.cc:654] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.883793 28074 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:17.884056 28071 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:17.884193 28071 ts_tablet_manager.cc:1367] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:17.884733 28070 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:17.884881 28070 ts_tablet_manager.cc:1367] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:46:17.884753 28071 raft_consensus.cc:348] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:17.885113 28071 raft_consensus.cc:374] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.885188 28071 raft_consensus.cc:729] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.885416 28071 consensus_queue.cc:260] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:17.885429 28070 raft_consensus.cc:348] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.885571 28070 raft_consensus.cc:374] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.885612 28070 raft_consensus.cc:729] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.885665 28074 tablet_bootstrap.cc:654] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.885766 28070 consensus_queue.cc:260] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.886190 28070 ts_tablet_manager.cc:1398] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:17.886433 28070 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:17.886955 28071 ts_tablet_manager.cc:1398] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:46:17.887495 28070 tablet_bootstrap.cc:654] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.888227 28072 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:17.888691 28070 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:17.888794 28070 ts_tablet_manager.cc:1367] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:17.889312 28070 raft_consensus.cc:348] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.889436 28070 raft_consensus.cc:374] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.889475 28070 raft_consensus.cc:729] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.889609 28070 consensus_queue.cc:260] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.889829 28072 tablet_bootstrap.cc:654] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.889914 28070 ts_tablet_manager.cc:1398] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:17.890115 28070 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:17.891119 28070 tablet_bootstrap.cc:654] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.892350 28070 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:17.892458 28070 ts_tablet_manager.cc:1367] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:17.892594 28074 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:17.892738 28074 ts_tablet_manager.cc:1367] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.000s
I20240502 20:46:17.892983 28070 raft_consensus.cc:348] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:17.893154 28070 raft_consensus.cc:374] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.893200 28070 raft_consensus.cc:729] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.893358 28070 consensus_queue.cc:260] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:17.893427 28074 raft_consensus.cc:348] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.893658 28074 raft_consensus.cc:374] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.893676 28070 ts_tablet_manager.cc:1398] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:17.893756 28074 raft_consensus.cc:729] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.893935 28074 consensus_queue.cc:260] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.894255 28074 ts_tablet_manager.cc:1398] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:17.894466 28074 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:17.895493 28072 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:17.895612 28072 ts_tablet_manager.cc:1367] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 20:46:17.896059 28072 raft_consensus.cc:348] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.896159 28072 raft_consensus.cc:374] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.896185 28072 raft_consensus.cc:729] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.896346 28072 consensus_queue.cc:260] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.897205 28072 ts_tablet_manager.cc:1398] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:46:17.897387 28072 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:17.898219 28074 tablet_bootstrap.cc:654] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.899597 28072 tablet_bootstrap.cc:654] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.900342 28074 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:17.900456 28074 ts_tablet_manager.cc:1367] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:46:17.901176 28074 raft_consensus.cc:348] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.901293 28074 raft_consensus.cc:374] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.901330 28074 raft_consensus.cc:729] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.901479 28074 consensus_queue.cc:260] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.901674 28072 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:17.901760 28074 ts_tablet_manager.cc:1398] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:17.901875 28072 ts_tablet_manager.cc:1367] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 20:46:17.901940 28074 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:17.902630 28072 raft_consensus.cc:348] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.902719 27984 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:17.902796 28072 raft_consensus.cc:374] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.902851 28072 raft_consensus.cc:729] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.902850 27984 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.903026 28074 tablet_bootstrap.cc:654] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:17.903316 28072 consensus_queue.cc:260] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.903517 27984 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:17.904191 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:17.904316 28072 ts_tablet_manager.cc:1398] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:17.904316 27362 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:17.905187 27026 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:17.905654 27984 raft_consensus.cc:2781] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:17.905726 27984 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:17.905776 27984 raft_consensus.cc:3037] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.906948 27984 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.907407 28074 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:17.907394 27984 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Requested vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:17.907514 28074 ts_tablet_manager.cc:1367] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:46:17.908061 28074 raft_consensus.cc:348] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:17.908195 28074 raft_consensus.cc:374] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:17.908164 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:17.908226 28074 raft_consensus.cc:729] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:17.908279 27362 raft_consensus.cc:3037] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.908368 28074 consensus_queue.cc:260] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:17.908672 28074 ts_tablet_manager.cc:1398] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:17.909376 27362 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:17.909444 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:17.909557 27228 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:17.909533 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:17.909693 27227 raft_consensus.cc:3037] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.909813 27026 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:17.910208 27984 raft_consensus.cc:2781] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:17.910387 27984 raft_consensus.cc:686] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Becoming Leader. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:17.910560 27984 consensus_queue.cc:237] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.910863 27227 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:17.912719 26801 catalog_manager.cc:5484] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:17.920774 27984 raft_consensus.cc:482] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:17.920943 27984 raft_consensus.cc:504] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.921591 27984 leader_election.cc:288] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:17.922192 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b9ff3c921d474fa5ad106d1c17ab00" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:17.922353 27227 raft_consensus.cc:2454] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:17.922433 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b9ff3c921d474fa5ad106d1c17ab00" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:17.922709 27362 raft_consensus.cc:2454] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:17.922820 27027 leader_election.cc:302] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:17.923034 27984 raft_consensus.cc:2781] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:17.923113 27984 raft_consensus.cc:482] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:17.923156 27984 raft_consensus.cc:3037] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.924146 27984 raft_consensus.cc:504] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.924539 27984 leader_election.cc:288] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Requested vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:17.924959 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b9ff3c921d474fa5ad106d1c17ab00" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:17.925011 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b9ff3c921d474fa5ad106d1c17ab00" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:17.925151 27227 raft_consensus.cc:3037] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.925173 27362 raft_consensus.cc:3037] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:17.926009 27227 raft_consensus.cc:2454] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:17.926014 27362 raft_consensus.cc:2454] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:17.926347 27026 leader_election.cc:302] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:17.926607 27984 raft_consensus.cc:2781] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:17.926688 27984 raft_consensus.cc:686] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Becoming Leader. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:17.926800 27984 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:17.928718 26801 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:18.251947 28043 raft_consensus.cc:482] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:18.252126 28043 raft_consensus.cc:504] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.252698 28043 leader_election.cc:288] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:18.253332 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b466a750cc6b4382947282d8aa6aeb1f" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:18.253501 26959 raft_consensus.cc:2454] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:18.253865 27158 leader_election.cc:302] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:18.254058 28043 raft_consensus.cc:2781] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:18.254122 28043 raft_consensus.cc:482] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:18.254149 28043 raft_consensus.cc:3037] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.255332 28043 raft_consensus.cc:504] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.255831 28043 leader_election.cc:288] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Requested vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:18.256402 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b466a750cc6b4382947282d8aa6aeb1f" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:18.256620 26959 raft_consensus.cc:3037] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.257977 26959 raft_consensus.cc:2454] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:18.257896 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b466a750cc6b4382947282d8aa6aeb1f" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:18.257898 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b466a750cc6b4382947282d8aa6aeb1f" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:18.258127 27093 raft_consensus.cc:2454] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:18.258353 27158 leader_election.cc:302] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:18.258574 28043 raft_consensus.cc:2781] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:18.258700 28043 raft_consensus.cc:686] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Leader. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:18.258860 28043 consensus_queue.cc:237] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.261309 26801 catalog_manager.cc:5484] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:18.349792 27984 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:18.355165 27984 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:18.424469 27984 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:18.428241 27984 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:46:18.480377 28088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:18.480698 28088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:18.484130 28088 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 20:46:18.611454 26959 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:18.612012 27093 raft_consensus.cc:3037] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.611971 28024 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:18.613399 27093 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:18.615473 28024 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:18.639780 28093 mvcc.cc:205] Tried to move back new op lower bound from 7023340661238767616 to 7023340659749969920. Current Snapshot: MvccSnapshot[applied={T|T < 7023340661187833856}]

INSERT report
    rows total: 2000
    time total: 155.415 ms
  time per row: 0.0777076 ms
I20240502 20:46:18.704039 26802 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47424:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:46:18.713419 27070 tablet_service.cc:1467] Processing CreateTablet for tablet f00c4b7954564820875a25da5834416a (DEFAULT_TABLE table=rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:18.713594 26936 tablet_service.cc:1467] Processing CreateTablet for tablet 843a26fc48844bfd82911b6a27833023 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:18.713815 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00c4b7954564820875a25da5834416a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.713748 26938 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8ce1df86914b2caf2d083d97eda4e6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:18.713922 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843a26fc48844bfd82911b6a27833023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.714090 26938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8ce1df86914b2caf2d083d97eda4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.714351 26937 tablet_service.cc:1467] Processing CreateTablet for tablet f00c4b7954564820875a25da5834416a (DEFAULT_TABLE table=rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:18.714588 26937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00c4b7954564820875a25da5834416a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.716441 28151 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:18.717337 27207 tablet_service.cc:1467] Processing CreateTablet for tablet 843a26fc48844bfd82911b6a27833023 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:18.717653 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843a26fc48844bfd82911b6a27833023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.717834 28151 tablet_bootstrap.cc:654] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.719666 28152 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:18.720526 28154 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:18.720976 28152 tablet_bootstrap.cc:654] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.721990 28154 tablet_bootstrap.cc:654] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.722700 28152 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:18.722829 28152 ts_tablet_manager.cc:1367] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:46:18.723385 28152 raft_consensus.cc:348] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:18.717285 27342 tablet_service.cc:1467] Processing CreateTablet for tablet f00c4b7954564820875a25da5834416a (DEFAULT_TABLE table=rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:18.723529 28152 raft_consensus.cc:374] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.723572 28152 raft_consensus.cc:729] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.722833 27340 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8ce1df86914b2caf2d083d97eda4e6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:18.723160 27341 tablet_service.cc:1467] Processing CreateTablet for tablet 843a26fc48844bfd82911b6a27833023 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:18.723739 28152 consensus_queue.cc:260] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:18.723923 27070 tablet_service.cc:1467] Processing CreateTablet for tablet 6a8ce1df86914b2caf2d083d97eda4e6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:18.724023 28152 ts_tablet_manager.cc:1398] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:18.724169 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8ce1df86914b2caf2d083d97eda4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.723990 27341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843a26fc48844bfd82911b6a27833023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.724490 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00c4b7954564820875a25da5834416a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.724772 27340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8ce1df86914b2caf2d083d97eda4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:18.726593 28152 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:18.729266 28152 tablet_bootstrap.cc:654] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.730748 28157 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:18.731094 28151 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:18.731233 28151 ts_tablet_manager.cc:1367] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.000s
I20240502 20:46:18.731838 28157 tablet_bootstrap.cc:654] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.731894 28151 raft_consensus.cc:348] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:18.732069 28151 raft_consensus.cc:374] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.732123 28151 raft_consensus.cc:729] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.732311 28151 consensus_queue.cc:260] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:18.732642 28154 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:18.732813 28154 ts_tablet_manager.cc:1367] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.000s
I20240502 20:46:18.733460 28151 ts_tablet_manager.cc:1398] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:18.733686 28157 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:18.733722 28151 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:18.733794 28157 ts_tablet_manager.cc:1367] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:18.734308 28157 raft_consensus.cc:348] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:18.734433 28157 raft_consensus.cc:374] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.734481 28157 raft_consensus.cc:729] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.734678 28157 consensus_queue.cc:260] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:18.735091 28157 ts_tablet_manager.cc:1398] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:18.735287 28157 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:18.736145 28152 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:18.736271 28152 ts_tablet_manager.cc:1367] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.003s
I20240502 20:46:18.737335 28152 raft_consensus.cc:348] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:18.737516 28152 raft_consensus.cc:374] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.737574 28152 raft_consensus.cc:729] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.737634 28157 tablet_bootstrap.cc:654] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.737941 28151 tablet_bootstrap.cc:654] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.738651 28154 raft_consensus.cc:348] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.738842 28154 raft_consensus.cc:374] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.738910 28154 raft_consensus.cc:729] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.739025 28152 consensus_queue.cc:260] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:18.739073 28154 consensus_queue.cc:260] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.739287 28157 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:18.739401 28157 ts_tablet_manager.cc:1367] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 20:46:18.739583 28152 ts_tablet_manager.cc:1398] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:46:18.739923 28157 raft_consensus.cc:348] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.740051 28157 raft_consensus.cc:374] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.740094 28157 raft_consensus.cc:729] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.740178 28154 ts_tablet_manager.cc:1398] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.001s
I20240502 20:46:18.740311 28157 consensus_queue.cc:260] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.740442 28022 raft_consensus.cc:482] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:18.740515 28022 raft_consensus.cc:504] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.741437 28151 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:18.741587 28151 ts_tablet_manager.cc:1367] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240502 20:46:18.741946 28022 leader_election.cc:288] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:18.742122 28151 raft_consensus.cc:348] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:18.742240 28151 raft_consensus.cc:374] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.742277 28151 raft_consensus.cc:729] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.742430 28157 ts_tablet_manager.cc:1398] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:46:18.742429 28151 consensus_queue.cc:260] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:18.742650 28157 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:18.742748 28151 ts_tablet_manager.cc:1398] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:18.742933 28151 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:18.743834 28157 tablet_bootstrap.cc:654] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.743968 28151 tablet_bootstrap.cc:654] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:18.745524 28157 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:18.745635 28157 ts_tablet_manager.cc:1367] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:46:18.746330 28157 raft_consensus.cc:348] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:18.746480 28157 raft_consensus.cc:374] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.746526 28157 raft_consensus.cc:729] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.746666 28157 consensus_queue.cc:260] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:18.747009 28157 ts_tablet_manager.cc:1398] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:18.747773 28151 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:18.747898 28151 ts_tablet_manager.cc:1367] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:46:18.748490 28151 raft_consensus.cc:348] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.748612 28151 raft_consensus.cc:374] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:18.748646 28151 raft_consensus.cc:729] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:18.748781 28151 consensus_queue.cc:260] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.749157 28151 ts_tablet_manager.cc:1398] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:18.751992 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843a26fc48844bfd82911b6a27833023" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:18.752205 26959 raft_consensus.cc:2454] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:18.752655 27295 leader_election.cc:302] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:18.752940 28022 raft_consensus.cc:2781] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:18.752987 28022 raft_consensus.cc:482] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:18.753013 28022 raft_consensus.cc:3037] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.754227 28022 raft_consensus.cc:504] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.754724 28022 leader_election.cc:288] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Requested vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:18.755218 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843a26fc48844bfd82911b6a27833023" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:18.755364 26959 raft_consensus.cc:3037] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.755581 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843a26fc48844bfd82911b6a27833023" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:18.755581 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843a26fc48844bfd82911b6a27833023" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:18.755789 27228 raft_consensus.cc:3037] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.756345 26959 raft_consensus.cc:2454] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:18.756631 27228 raft_consensus.cc:2454] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:18.756736 27295 leader_election.cc:302] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:18.756953 28022 raft_consensus.cc:2781] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:18.757246 28022 raft_consensus.cc:686] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Becoming Leader. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Running, Role: LEADER
I20240502 20:46:18.757594 28022 consensus_queue.cc:237] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:18.760783 26802 catalog_manager.cc:5484] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132). New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:18.856410 28130 raft_consensus.cc:482] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:18.856586 28130 raft_consensus.cc:504] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:18.857285 28130 leader_election.cc:288] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:18.857837 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00c4b7954564820875a25da5834416a" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:18.857870 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00c4b7954564820875a25da5834416a" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:18.858021 26959 raft_consensus.cc:2454] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:18.858059 27362 raft_consensus.cc:2454] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:18.858526 27024 leader_election.cc:302] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:18.858781 28130 raft_consensus.cc:2781] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:18.858865 28130 raft_consensus.cc:482] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:18.858911 28130 raft_consensus.cc:3037] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.859963 28130 raft_consensus.cc:504] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:18.860388 28130 leader_election.cc:288] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Requested vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:18.860906 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00c4b7954564820875a25da5834416a" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:18.860906 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00c4b7954564820875a25da5834416a" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:18.861186 27362 raft_consensus.cc:3037] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.861245 26959 raft_consensus.cc:3037] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.862785 27362 raft_consensus.cc:2454] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:18.862785 26959 raft_consensus.cc:2454] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:18.863413 27026 leader_election.cc:302] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:18.863787 28130 raft_consensus.cc:2781] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:18.863921 28130 raft_consensus.cc:686] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Becoming Leader. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:18.864159 28130 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:18.866828 26802 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:18.903690 28022 raft_consensus.cc:482] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:18.903890 28022 raft_consensus.cc:504] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:18.904495 28022 leader_election.cc:288] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:18.905472 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8ce1df86914b2caf2d083d97eda4e6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:18.905709 26959 raft_consensus.cc:2454] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:18.906080 27295 leader_election.cc:302] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:18.906287 28022 raft_consensus.cc:2781] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:18.906345 28022 raft_consensus.cc:482] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:18.906396 28022 raft_consensus.cc:3037] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.907379 28022 raft_consensus.cc:504] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:18.907824 28022 leader_election.cc:288] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Requested vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:18.908335 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8ce1df86914b2caf2d083d97eda4e6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:18.908578 26959 raft_consensus.cc:3037] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.909734 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8ce1df86914b2caf2d083d97eda4e6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:18.909739 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a8ce1df86914b2caf2d083d97eda4e6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:18.909933 27093 raft_consensus.cc:3037] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:18.909955 26959 raft_consensus.cc:2454] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:18.910403 27295 leader_election.cc:302] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:18.910635 28022 raft_consensus.cc:2781] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:18.910845 28022 raft_consensus.cc:686] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Becoming Leader. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Running, Role: LEADER
I20240502 20:46:18.911033 27093 raft_consensus.cc:2454] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:18.911051 28022 consensus_queue.cc:237] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:18.913233 26802 catalog_manager.cc:5484] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132). New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } }
W20240502 20:46:19.050534 28171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:19.050843 28171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:19.054563 28171 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 20:46:19.157999 27362 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:19.158279 26959 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:19.158599 28095 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:19.158756 28130 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:19.169885 26960 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:19.170615 28167 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:19.171509 27228 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:19.172808 26957 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:19.173945 27093 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:19.174116 28022 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:19.174527 28207 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:19.176853 28167 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 137.437 ms
  time per row: 0.0687183 ms
I20240502 20:46:19.247454 26805 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47424:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:46:19.254691 27207 tablet_service.cc:1467] Processing CreateTablet for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba (DEFAULT_TABLE table=rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:19.255138 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:19.255036 27070 tablet_service.cc:1467] Processing CreateTablet for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba (DEFAULT_TABLE table=rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:19.255308 26936 tablet_service.cc:1467] Processing CreateTablet for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba (DEFAULT_TABLE table=rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:19.255534 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:19.255597 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:19.257970 28226 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:19.258041 27204 tablet_service.cc:1467] Processing CreateTablet for tablet a2b61909194d488682afd91c0f796405 (DEFAULT_TABLE table=rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:19.258190 27342 tablet_service.cc:1467] Processing CreateTablet for tablet a2b61909194d488682afd91c0f796405 (DEFAULT_TABLE table=rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:19.258308 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b61909194d488682afd91c0f796405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:19.258515 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b61909194d488682afd91c0f796405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:19.258997 28226 tablet_bootstrap.cc:654] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:19.259016 28225 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:19.259981 28225 tablet_bootstrap.cc:654] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:19.261279 28225 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:19.261399 28225 ts_tablet_manager.cc:1367] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:19.261356 26937 tablet_service.cc:1467] Processing CreateTablet for tablet a2b61909194d488682afd91c0f796405 (DEFAULT_TABLE table=rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:19.261654 26937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b61909194d488682afd91c0f796405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:19.261858 28226 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:19.261958 28226 ts_tablet_manager.cc:1367] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:46:19.261952 28225 raft_consensus.cc:348] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:19.262096 28225 raft_consensus.cc:374] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:19.262138 28225 raft_consensus.cc:729] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:19.262315 28225 consensus_queue.cc:260] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:19.262462 28226 raft_consensus.cc:348] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:19.262594 28226 raft_consensus.cc:374] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:19.262638 28226 raft_consensus.cc:729] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:19.262640 28225 ts_tablet_manager.cc:1398] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:19.262785 28226 consensus_queue.cc:260] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:19.263069 28226 ts_tablet_manager.cc:1398] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:19.263094 28230 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:19.264161 28230 tablet_bootstrap.cc:654] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:19.265380 27070 tablet_service.cc:1467] Processing CreateTablet for tablet 5586b9e60b544aed87c24f7b3ce0198c (DEFAULT_TABLE table=rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:19.265535 28226 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:19.265700 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5586b9e60b544aed87c24f7b3ce0198c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:19.266379 27342 tablet_service.cc:1467] Processing CreateTablet for tablet 5586b9e60b544aed87c24f7b3ce0198c (DEFAULT_TABLE table=rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:19.266736 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5586b9e60b544aed87c24f7b3ce0198c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:19.267171 27204 tablet_service.cc:1467] Processing CreateTablet for tablet 5586b9e60b544aed87c24f7b3ce0198c (DEFAULT_TABLE table=rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:19.267470 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5586b9e60b544aed87c24f7b3ce0198c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:19.268353 28225 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:19.269455 28225 tablet_bootstrap.cc:654] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:19.270109 28226 tablet_bootstrap.cc:654] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:19.270885 28227 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:19.270920 28225 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:19.271129 28225 ts_tablet_manager.cc:1367] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:19.271330 28230 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:19.271445 28230 ts_tablet_manager.cc:1367] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 20:46:19.271885 28227 tablet_bootstrap.cc:654] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:19.271845 28225 raft_consensus.cc:348] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:19.271973 28225 raft_consensus.cc:374] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:19.272014 28225 raft_consensus.cc:729] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:19.272013 28230 raft_consensus.cc:348] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:19.272153 28230 raft_consensus.cc:374] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:19.272197 28230 raft_consensus.cc:729] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:19.272171 28225 consensus_queue.cc:260] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:19.272379 28230 consensus_queue.cc:260] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:19.272485 28225 ts_tablet_manager.cc:1398] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:19.272787 28230 ts_tablet_manager.cc:1398] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:19.272987 28230 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:19.273808 28227 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:19.273903 28227 ts_tablet_manager.cc:1367] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:19.274081 28230 tablet_bootstrap.cc:654] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:19.274411 28227 raft_consensus.cc:348] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:19.274539 28227 raft_consensus.cc:374] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:19.274577 28227 raft_consensus.cc:729] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:19.274760 28227 consensus_queue.cc:260] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:19.275334 28226 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:19.275449 28226 ts_tablet_manager.cc:1367] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 20:46:19.276018 28227 ts_tablet_manager.cc:1398] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:46:19.276212 28227 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:19.276154 28226 raft_consensus.cc:348] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:19.276278 28226 raft_consensus.cc:374] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:19.276329 28226 raft_consensus.cc:729] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:19.276465 28226 consensus_queue.cc:260] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:19.276782 28226 ts_tablet_manager.cc:1398] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:19.277524 28227 tablet_bootstrap.cc:654] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:19.277988 28230 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:19.278092 28230 ts_tablet_manager.cc:1367] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:46:19.278816 28227 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:19.278816 28230 raft_consensus.cc:348] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:19.279035 28230 raft_consensus.cc:374] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:19.279032 28227 ts_tablet_manager.cc:1367] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:19.279070 28230 raft_consensus.cc:729] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:19.279320 28230 consensus_queue.cc:260] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:19.279872 28230 ts_tablet_manager.cc:1398] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 20:46:19.279804 28227 raft_consensus.cc:348] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:19.279963 28227 raft_consensus.cc:374] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:19.280009 28227 raft_consensus.cc:729] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:19.280211 28227 consensus_queue.cc:260] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:19.280567 28227 ts_tablet_manager.cc:1398] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:19.280797 28227 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:19.282169 28227 tablet_bootstrap.cc:654] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:19.283708 28227 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:19.283813 28227 ts_tablet_manager.cc:1367] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:19.284287 28227 raft_consensus.cc:348] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:19.284405 28227 raft_consensus.cc:374] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:19.284442 28227 raft_consensus.cc:729] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:19.284641 28227 consensus_queue.cc:260] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:19.284968 28227 ts_tablet_manager.cc:1398] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:19.292927 28024 raft_consensus.cc:482] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:19.293144 28024 raft_consensus.cc:504] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:19.293668 28024 leader_election.cc:288] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:19.294205 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c71a33fbcc4cfdbf0065a3fc9dfeba" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:19.294209 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c71a33fbcc4cfdbf0065a3fc9dfeba" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:19.294359 26957 raft_consensus.cc:2454] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:19.294375 27093 raft_consensus.cc:2454] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:19.294867 27161 leader_election.cc:302] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:19.295125 28024 raft_consensus.cc:2781] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:19.295215 28024 raft_consensus.cc:482] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:19.295257 28024 raft_consensus.cc:3037] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:19.296445 28024 raft_consensus.cc:504] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:19.296870 28024 leader_election.cc:288] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Requested vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:19.297436 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c71a33fbcc4cfdbf0065a3fc9dfeba" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:19.297425 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c71a33fbcc4cfdbf0065a3fc9dfeba" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:19.297565 27093 raft_consensus.cc:3037] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:19.297582 26957 raft_consensus.cc:3037] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:19.298434 27093 raft_consensus.cc:2454] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:19.298681 26957 raft_consensus.cc:2454] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:19.298796 27161 leader_election.cc:302] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:19.299019 28024 raft_consensus.cc:2781] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:19.299286 28024 raft_consensus.cc:686] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Leader. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:19.299464 28024 consensus_queue.cc:237] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:19.301963 26801 catalog_manager.cc:5484] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:19.353302 28045 raft_consensus.cc:482] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:19.353497 28045 raft_consensus.cc:504] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:19.354218 28045 leader_election.cc:288] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:19.355134 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b61909194d488682afd91c0f796405" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:19.355331 27228 raft_consensus.cc:2454] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 0.
I20240502 20:46:19.355688 26894 leader_election.cc:302] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:19.355870 28045 raft_consensus.cc:2781] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:19.355922 28045 raft_consensus.cc:482] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:19.355957 28045 raft_consensus.cc:3037] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:19.357168 28045 raft_consensus.cc:504] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:19.357602 28045 leader_election.cc:288] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 election: Requested vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:19.357985 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b61909194d488682afd91c0f796405" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:19.358112 27228 raft_consensus.cc:3037] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:19.359072 27228 raft_consensus.cc:2454] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 1.
I20240502 20:46:19.359417 26894 leader_election.cc:302] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:19.359717 28045 raft_consensus.cc:2781] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:19.359714 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b61909194d488682afd91c0f796405" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:19.359745 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b61909194d488682afd91c0f796405" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:19.359891 27362 raft_consensus.cc:2454] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 0.
I20240502 20:46:19.359930 28045 raft_consensus.cc:686] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Becoming Leader. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Running, Role: LEADER
I20240502 20:46:19.360076 28045 consensus_queue.cc:237] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:19.362257 26801 catalog_manager.cc:5484] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297de823852d4a8da44791ab03131cd3 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:19.640785 28207 raft_consensus.cc:482] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:19.640945 28207 raft_consensus.cc:504] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:19.641439 28207 leader_election.cc:288] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:19.642060 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5586b9e60b544aed87c24f7b3ce0198c" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:19.642226 27228 raft_consensus.cc:2454] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:19.642062 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5586b9e60b544aed87c24f7b3ce0198c" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:19.642362 27093 raft_consensus.cc:2454] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:19.642627 27298 leader_election.cc:302] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:19.642817 28207 raft_consensus.cc:2781] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:19.642889 28207 raft_consensus.cc:482] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:19.642937 28207 raft_consensus.cc:3037] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:19.644155 28207 raft_consensus.cc:504] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:19.644543 28207 leader_election.cc:288] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Requested vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:19.645283 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5586b9e60b544aed87c24f7b3ce0198c" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:19.645447 27093 raft_consensus.cc:3037] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:19.645407 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5586b9e60b544aed87c24f7b3ce0198c" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:19.645563 27228 raft_consensus.cc:3037] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:19.646483 27228 raft_consensus.cc:2454] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:19.646827 27093 raft_consensus.cc:2454] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:19.647010 27298 leader_election.cc:302] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:19.647380 28207 raft_consensus.cc:2781] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:19.647631 28207 raft_consensus.cc:686] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Becoming Leader. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Running, Role: LEADER
I20240502 20:46:19.647812 28207 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:19.650019 26801 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132). New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:19.707662 28024 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:19.711388 28024 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:19.760588 27361 raft_consensus.cc:3037] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:19.762931 28241 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:19.766724 28249 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:46:19.851897 28247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:19.852269 28247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:19.855957 28247 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 20:46:19.970572 27228 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:19.972210 27092 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:19.972976 28022 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:19.977559 28207 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 137.486 ms
  time per row: 0.0687432 ms
I20240502 20:46:20.049928 26802 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47424:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:46:20.057677 27070 tablet_service.cc:1467] Processing CreateTablet for tablet 84484d65e2bb4e23ac621a39434ce382 (DEFAULT_TABLE table=rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.057997 27342 tablet_service.cc:1467] Processing CreateTablet for tablet 14a7bead40fb4eb0bec6b00804b52595 (DEFAULT_TABLE table=rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.057972 27204 tablet_service.cc:1467] Processing CreateTablet for tablet 84484d65e2bb4e23ac621a39434ce382 (DEFAULT_TABLE table=rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.058149 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84484d65e2bb4e23ac621a39434ce382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.058382 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a7bead40fb4eb0bec6b00804b52595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.058466 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84484d65e2bb4e23ac621a39434ce382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.061235 28310 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:20.061524 26937 tablet_service.cc:1467] Processing CreateTablet for tablet 84484d65e2bb4e23ac621a39434ce382 (DEFAULT_TABLE table=rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.061877 26937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84484d65e2bb4e23ac621a39434ce382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.061821 27341 tablet_service.cc:1467] Processing CreateTablet for tablet 8deaa1f751034c15acd6ccaad37536c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.062116 27341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8deaa1f751034c15acd6ccaad37536c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.062848 28310 tablet_bootstrap.cc:654] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.064330 28310 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:20.064438 28310 ts_tablet_manager.cc:1367] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:46:20.064863 27070 tablet_service.cc:1467] Processing CreateTablet for tablet 14a7bead40fb4eb0bec6b00804b52595 (DEFAULT_TABLE table=rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.065025 27207 tablet_service.cc:1467] Processing CreateTablet for tablet 8deaa1f751034c15acd6ccaad37536c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.065382 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8deaa1f751034c15acd6ccaad37536c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.065515 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a7bead40fb4eb0bec6b00804b52595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.066228 26936 tablet_service.cc:1467] Processing CreateTablet for tablet 8deaa1f751034c15acd6ccaad37536c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.066363 28311 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:20.066515 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8deaa1f751034c15acd6ccaad37536c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.067379 28311 tablet_bootstrap.cc:654] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.068465 28311 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:20.068572 28311 ts_tablet_manager.cc:1367] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:20.068696 28310 raft_consensus.cc:348] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:20.068949 28310 raft_consensus.cc:374] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.069054 28311 raft_consensus.cc:348] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:20.069227 28311 raft_consensus.cc:374] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.069274 28311 raft_consensus.cc:729] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.069137 28310 raft_consensus.cc:729] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.069408 28311 consensus_queue.cc:260] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:20.069531 28310 consensus_queue.cc:260] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:20.069718 28311 ts_tablet_manager.cc:1398] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:46:20.069902 28311 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:20.070959 28311 tablet_bootstrap.cc:654] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.071050 28310 ts_tablet_manager.cc:1398] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240502 20:46:20.071254 28310 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:20.072557 28310 tablet_bootstrap.cc:654] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.072667 26937 tablet_service.cc:1467] Processing CreateTablet for tablet 14a7bead40fb4eb0bec6b00804b52595 (DEFAULT_TABLE table=rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.073459 26937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a7bead40fb4eb0bec6b00804b52595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.074129 28315 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:20.075436 28315 tablet_bootstrap.cc:654] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.077298 28314 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:20.077484 28315 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:20.077584 28315 ts_tablet_manager.cc:1367] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:46:20.078657 28315 raft_consensus.cc:348] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:20.078858 28315 raft_consensus.cc:374] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.078905 28315 raft_consensus.cc:729] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.078965 28314 tablet_bootstrap.cc:654] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.079224 28315 consensus_queue.cc:260] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:20.079663 28315 ts_tablet_manager.cc:1398] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 20:46:20.079901 28315 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:20.080634 28314 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:20.080771 28314 ts_tablet_manager.cc:1367] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 20:46:20.080986 28311 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:20.081135 28311 ts_tablet_manager.cc:1367] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.011s	user 0.001s	sys 0.002s
I20240502 20:46:20.081251 28315 tablet_bootstrap.cc:654] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.081383 28314 raft_consensus.cc:348] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:20.081526 28314 raft_consensus.cc:374] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.081566 28314 raft_consensus.cc:729] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.081671 28311 raft_consensus.cc:348] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:20.081790 28311 raft_consensus.cc:374] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.081838 28311 raft_consensus.cc:729] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.082010 28311 consensus_queue.cc:260] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:20.082114 28314 consensus_queue.cc:260] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:20.082502 28310 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:20.082603 28310 ts_tablet_manager.cc:1367] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.000s
I20240502 20:46:20.083035 28311 ts_tablet_manager.cc:1398] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:46:20.083151 28310 raft_consensus.cc:348] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.083266 28310 raft_consensus.cc:374] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.083308 28310 raft_consensus.cc:729] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.083462 28310 consensus_queue.cc:260] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.083815 28310 ts_tablet_manager.cc:1398] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:20.084594 28315 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:20.084745 28315 ts_tablet_manager.cc:1367] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 20:46:20.085315 28315 raft_consensus.cc:348] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.085456 28315 raft_consensus.cc:374] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.085450 28314 ts_tablet_manager.cc:1398] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:46:20.085494 28315 raft_consensus.cc:729] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.085630 28315 consensus_queue.cc:260] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.085937 28315 ts_tablet_manager.cc:1398] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:20.086848 28314 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:20.088060 28314 tablet_bootstrap.cc:654] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.089594 28314 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:20.089722 28314 ts_tablet_manager.cc:1367] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:20.090159 28314 raft_consensus.cc:348] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.090276 28314 raft_consensus.cc:374] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.090304 28314 raft_consensus.cc:729] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.090409 28314 consensus_queue.cc:260] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.090694 28314 ts_tablet_manager.cc:1398] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:20.090903 28314 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:20.091842 28314 tablet_bootstrap.cc:654] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.098574 28314 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:20.098742 28314 ts_tablet_manager.cc:1367] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 20:46:20.099378 28314 raft_consensus.cc:348] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:20.099521 28314 raft_consensus.cc:374] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.099582 28314 raft_consensus.cc:729] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.099735 28314 consensus_queue.cc:260] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:20.100064 28314 ts_tablet_manager.cc:1398] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:20.324379 28291 raft_consensus.cc:482] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:20.324568 28291 raft_consensus.cc:504] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:20.325146 28291 leader_election.cc:288] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:20.325686 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84484d65e2bb4e23ac621a39434ce382" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:20.325699 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84484d65e2bb4e23ac621a39434ce382" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:20.325842 27092 raft_consensus.cc:2454] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:20.325887 26957 raft_consensus.cc:2454] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:20.326231 27161 leader_election.cc:302] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:20.326426 28291 raft_consensus.cc:2781] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:20.326496 28291 raft_consensus.cc:482] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:20.326531 28291 raft_consensus.cc:3037] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.327459 28291 raft_consensus.cc:504] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:20.327858 28291 leader_election.cc:288] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Requested vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:20.328390 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84484d65e2bb4e23ac621a39434ce382" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:20.328392 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84484d65e2bb4e23ac621a39434ce382" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:20.328562 26957 raft_consensus.cc:3037] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.328560 27092 raft_consensus.cc:3037] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.330366 26957 raft_consensus.cc:2454] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:20.330366 27092 raft_consensus.cc:2454] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:20.331012 27161 leader_election.cc:302] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:20.331342 28291 raft_consensus.cc:2781] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:20.331452 28291 raft_consensus.cc:686] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Leader. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:20.331619 28291 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:20.334167 26809 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:20.339277 28022 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:20.339461 28022 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:20.340031 28022 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:20.340617 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:20.340823 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:20.340986 27092 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:20.340996 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:20.341564 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:20.341923 28022 raft_consensus.cc:2781] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:20.341985 28022 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:20.342031 28022 raft_consensus.cc:3037] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.343073 28022 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:20.343523 28022 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Requested vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:20.344050 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:20.344125 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:20.344193 27092 raft_consensus.cc:3037] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.344307 26959 raft_consensus.cc:3037] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.345283 27092 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:20.345522 26959 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:20.345728 27298 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:20.346017 28022 raft_consensus.cc:2781] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:20.346135 28022 raft_consensus.cc:686] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Becoming Leader. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Running, Role: LEADER
I20240502 20:46:20.346529 28022 consensus_queue.cc:237] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:20.348740 26809 catalog_manager.cc:5484] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132). New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:20.359349 28022 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:20.359516 28022 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.360005 28022 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:20.360611 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8deaa1f751034c15acd6ccaad37536c6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:20.360613 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8deaa1f751034c15acd6ccaad37536c6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:20.360765 27228 raft_consensus.cc:2454] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:20.360780 26959 raft_consensus.cc:2454] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:20.361152 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:20.361351 28022 raft_consensus.cc:2781] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:20.361415 28022 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:20.361444 28022 raft_consensus.cc:3037] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.362354 28022 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.362782 28022 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Requested vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:20.363240 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8deaa1f751034c15acd6ccaad37536c6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:20.363243 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8deaa1f751034c15acd6ccaad37536c6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:20.363391 27228 raft_consensus.cc:3037] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.363588 26959 raft_consensus.cc:3037] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.364351 27228 raft_consensus.cc:2454] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:20.364773 27298 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:20.364787 26959 raft_consensus.cc:2454] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:20.365006 28022 raft_consensus.cc:2781] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:20.365226 28022 raft_consensus.cc:686] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Becoming Leader. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Running, Role: LEADER
I20240502 20:46:20.365396 28022 consensus_queue.cc:237] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.367431 26809 catalog_manager.cc:5484] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132). New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } }
W20240502 20:46:20.592381 28327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:20.592753 28327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:20.596724 28327 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 20:46:20.729048 26959 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:20.729620 28291 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:20.731631 27092 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:20.732859 28291 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:20.734823 27093 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:20.735353 28022 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:20.739523 27228 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:20.740342 28022 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:20.742789 26957 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:20.745157 28022 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:20.745611 26959 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:20.746320 28207 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 157.608 ms
  time per row: 0.0788038 ms
I20240502 20:46:20.818554 26801 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47424:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:46:20.826056 27070 tablet_service.cc:1467] Processing CreateTablet for tablet bf189c19164442e5af0827f1fb7dc085 (DEFAULT_TABLE table=rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.826565 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf189c19164442e5af0827f1fb7dc085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.826612 27204 tablet_service.cc:1467] Processing CreateTablet for tablet bf189c19164442e5af0827f1fb7dc085 (DEFAULT_TABLE table=rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.826781 27207 tablet_service.cc:1467] Processing CreateTablet for tablet a906569ab3b14520b4876bc8d5450f7d (DEFAULT_TABLE table=rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.826898 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf189c19164442e5af0827f1fb7dc085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.827075 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a906569ab3b14520b4876bc8d5450f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.827337 26937 tablet_service.cc:1467] Processing CreateTablet for tablet fdd2311ec0b645f5a79098a32c6deba2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.827637 26937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd2311ec0b645f5a79098a32c6deba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.829530 28392 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:20.830109 27342 tablet_service.cc:1467] Processing CreateTablet for tablet bf189c19164442e5af0827f1fb7dc085 (DEFAULT_TABLE table=rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.830427 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf189c19164442e5af0827f1fb7dc085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.830960 28392 tablet_bootstrap.cc:654] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.831372 27070 tablet_service.cc:1467] Processing CreateTablet for tablet fdd2311ec0b645f5a79098a32c6deba2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.831779 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd2311ec0b645f5a79098a32c6deba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.831835 27207 tablet_service.cc:1467] Processing CreateTablet for tablet fdd2311ec0b645f5a79098a32c6deba2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.832131 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd2311ec0b645f5a79098a32c6deba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.833024 28396 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:20.829495 27071 tablet_service.cc:1467] Processing CreateTablet for tablet a906569ab3b14520b4876bc8d5450f7d (DEFAULT_TABLE table=rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.833128 27341 tablet_service.cc:1467] Processing CreateTablet for tablet a906569ab3b14520b4876bc8d5450f7d (DEFAULT_TABLE table=rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:20.833428 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a906569ab3b14520b4876bc8d5450f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.833645 27341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a906569ab3b14520b4876bc8d5450f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:20.836089 28394 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:20.836262 28393 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:20.837189 28394 tablet_bootstrap.cc:654] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.837603 28393 tablet_bootstrap.cc:654] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.837859 28392 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:20.838054 28392 ts_tablet_manager.cc:1367] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.000s
I20240502 20:46:20.838553 28394 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:20.838680 28394 ts_tablet_manager.cc:1367] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:46:20.838933 28392 raft_consensus.cc:348] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.839068 28396 tablet_bootstrap.cc:654] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.839272 28392 raft_consensus.cc:374] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.839272 28394 raft_consensus.cc:348] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:20.839435 28394 raft_consensus.cc:374] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.839473 28394 raft_consensus.cc:729] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.839625 28394 consensus_queue.cc:260] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:20.839990 28394 ts_tablet_manager.cc:1398] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
I20240502 20:46:20.839963 28392 raft_consensus.cc:729] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.840109 28393 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:20.840227 28393 ts_tablet_manager.cc:1367] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:46:20.840396 28396 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:20.840502 28396 ts_tablet_manager.cc:1367] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.002s
I20240502 20:46:20.840436 28392 consensus_queue.cc:260] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.840816 28393 raft_consensus.cc:348] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.840950 28393 raft_consensus.cc:374] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.840991 28393 raft_consensus.cc:729] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.840996 28396 raft_consensus.cc:348] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.841146 28396 raft_consensus.cc:374] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.841202 28396 raft_consensus.cc:729] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.841187 28393 consensus_queue.cc:260] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.841333 28396 consensus_queue.cc:260] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.841522 28393 ts_tablet_manager.cc:1398] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:20.841712 28392 ts_tablet_manager.cc:1398] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240502 20:46:20.841738 28393 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:20.841809 28396 ts_tablet_manager.cc:1398] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:20.842010 28392 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:20.842026 28396 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:20.842878 28393 tablet_bootstrap.cc:654] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.843153 28392 tablet_bootstrap.cc:654] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.843647 28396 tablet_bootstrap.cc:654] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.844563 28393 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:20.844758 28393 ts_tablet_manager.cc:1367] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:20.845399 28392 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:20.845506 28392 ts_tablet_manager.cc:1367] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 20:46:20.845366 28393 raft_consensus.cc:348] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.845654 28393 raft_consensus.cc:374] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.845747 28393 raft_consensus.cc:729] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.845928 28393 consensus_queue.cc:260] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.846058 28392 raft_consensus.cc:348] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:20.846194 28392 raft_consensus.cc:374] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.846254 28392 raft_consensus.cc:729] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.846414 28396 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:20.846434 28392 consensus_queue.cc:260] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:20.846530 28396 ts_tablet_manager.cc:1367] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 20:46:20.846709 28393 ts_tablet_manager.cc:1398] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:46:20.846879 28392 ts_tablet_manager.cc:1398] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.001s
I20240502 20:46:20.846982 28393 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:20.847091 28392 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:20.847048 28396 raft_consensus.cc:348] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.847152 28396 raft_consensus.cc:374] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.847208 28396 raft_consensus.cc:729] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.847357 28396 consensus_queue.cc:260] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.847673 28396 ts_tablet_manager.cc:1398] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.001s
I20240502 20:46:20.848206 28392 tablet_bootstrap.cc:654] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.849027 28393 tablet_bootstrap.cc:654] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:20.849686 28392 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:20.849784 28392 ts_tablet_manager.cc:1367] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 20:46:20.850291 28392 raft_consensus.cc:348] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.850430 28392 raft_consensus.cc:374] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.850472 28392 raft_consensus.cc:729] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.850606 28392 consensus_queue.cc:260] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.850939 28392 ts_tablet_manager.cc:1398] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:20.851408 28393 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:20.851507 28393 ts_tablet_manager.cc:1367] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 20:46:20.852057 28393 raft_consensus.cc:348] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:20.852181 28393 raft_consensus.cc:374] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:20.852222 28393 raft_consensus.cc:729] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:20.852358 28393 consensus_queue.cc:260] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:20.852659 28393 ts_tablet_manager.cc:1398] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:46:20.882082 28370 raft_consensus.cc:482] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:20.882298 28370 raft_consensus.cc:504] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.883041 28370 leader_election.cc:288] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:20.883567 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a906569ab3b14520b4876bc8d5450f7d" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:20.883579 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a906569ab3b14520b4876bc8d5450f7d" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:20.883718 27093 raft_consensus.cc:2454] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:20.883780 27361 raft_consensus.cc:2454] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:20.884068 27161 leader_election.cc:302] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:20.884406 28370 raft_consensus.cc:2781] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:20.884482 28370 raft_consensus.cc:482] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:20.884532 28370 raft_consensus.cc:3037] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.885665 28370 raft_consensus.cc:504] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.886087 28370 leader_election.cc:288] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Requested vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:20.886518 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a906569ab3b14520b4876bc8d5450f7d" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:20.886514 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a906569ab3b14520b4876bc8d5450f7d" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:20.886659 27361 raft_consensus.cc:3037] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.886662 27093 raft_consensus.cc:3037] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:20.887610 27093 raft_consensus.cc:2454] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:20.887804 27361 raft_consensus.cc:2454] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:20.887969 27161 leader_election.cc:302] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:20.888185 28370 raft_consensus.cc:2781] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:20.888284 28370 raft_consensus.cc:686] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Leader. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:20.888449 28370 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:20.891356 26801 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:21.061192 28370 raft_consensus.cc:482] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:21.061398 28370 raft_consensus.cc:504] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.061924 28370 leader_election.cc:288] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:21.062458 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf189c19164442e5af0827f1fb7dc085" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:21.062460 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf189c19164442e5af0827f1fb7dc085" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:21.062618 27361 raft_consensus.cc:2454] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:21.062626 27093 raft_consensus.cc:2454] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:21.062963 27161 leader_election.cc:302] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:21.063149 28370 raft_consensus.cc:2781] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:21.063206 28370 raft_consensus.cc:482] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:21.063234 28370 raft_consensus.cc:3037] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.074571 28370 raft_consensus.cc:504] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.075270 28370 leader_election.cc:288] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Requested vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:21.075980 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf189c19164442e5af0827f1fb7dc085" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:21.075980 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf189c19164442e5af0827f1fb7dc085" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:21.076179 27361 raft_consensus.cc:3037] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.076174 27093 raft_consensus.cc:3037] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.077229 27093 raft_consensus.cc:2454] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:21.077239 27361 raft_consensus.cc:2454] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:21.077721 27160 leader_election.cc:302] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:21.078006 28370 raft_consensus.cc:2781] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:21.078109 28370 raft_consensus.cc:686] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Leader. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:21.078295 28370 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.080528 26801 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:21.091985 28095 raft_consensus.cc:482] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:21.092180 28095 raft_consensus.cc:504] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:21.092706 28095 leader_election.cc:288] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:21.093297 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd2311ec0b645f5a79098a32c6deba2" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:21.093477 26957 raft_consensus.cc:2454] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:21.093515 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd2311ec0b645f5a79098a32c6deba2" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:21.093672 27228 raft_consensus.cc:2454] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:21.093993 27027 leader_election.cc:302] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:21.094296 28095 raft_consensus.cc:2781] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:21.094414 28095 raft_consensus.cc:482] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:21.094448 28095 raft_consensus.cc:3037] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.095563 28095 raft_consensus.cc:504] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:21.096087 28095 leader_election.cc:288] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Requested vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:21.096534 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd2311ec0b645f5a79098a32c6deba2" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:21.096697 26957 raft_consensus.cc:3037] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.096695 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd2311ec0b645f5a79098a32c6deba2" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:21.096818 27228 raft_consensus.cc:3037] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.097883 26957 raft_consensus.cc:2454] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:21.097883 27228 raft_consensus.cc:2454] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:21.098336 27024 leader_election.cc:302] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:21.098533 28095 raft_consensus.cc:2781] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:21.098709 28095 raft_consensus.cc:686] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Becoming Leader. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:21.099100 28095 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:21.101188 26801 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:21.281661 28409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:21.281991 28409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:21.285892 28409 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 20:46:21.381790 28238 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:21.387542 28370 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:21.399820 27093 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:21.399870 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:21.400549 28238 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:21.400897 28238 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:21.402781 26959 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:21.403421 27228 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:21.403913 28407 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:21.404091 28095 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 138.07 ms
  time per row: 0.0690349 ms
I20240502 20:46:21.481884 26807 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47424:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:46:21.490226 27341 tablet_service.cc:1467] Processing CreateTablet for tablet ff9fef57f6eb490da536170a02b96212 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:21.490413 27342 tablet_service.cc:1467] Processing CreateTablet for tablet b5f81e46ab1f48fc8a88495b42f83640 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:21.490414 26937 tablet_service.cc:1467] Processing CreateTablet for tablet ff9fef57f6eb490da536170a02b96212 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:21.490640 27341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9fef57f6eb490da536170a02b96212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:21.490715 26937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9fef57f6eb490da536170a02b96212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:21.491173 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f81e46ab1f48fc8a88495b42f83640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:21.491376 27071 tablet_service.cc:1467] Processing CreateTablet for tablet ff9fef57f6eb490da536170a02b96212 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:21.491415 27070 tablet_service.cc:1467] Processing CreateTablet for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:21.491732 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9fef57f6eb490da536170a02b96212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:21.492080 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:21.492012 27340 tablet_service.cc:1467] Processing CreateTablet for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:21.492178 27207 tablet_service.cc:1467] Processing CreateTablet for tablet b5f81e46ab1f48fc8a88495b42f83640 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:21.492261 27340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:21.492462 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f81e46ab1f48fc8a88495b42f83640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:21.494307 28465 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:21.494987 28466 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:21.495467 28465 tablet_bootstrap.cc:654] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:21.496134 26936 tablet_service.cc:1467] Processing CreateTablet for tablet b5f81e46ab1f48fc8a88495b42f83640 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:21.496207 26938 tablet_service.cc:1467] Processing CreateTablet for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:21.496399 28466 tablet_bootstrap.cc:654] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:21.496521 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f81e46ab1f48fc8a88495b42f83640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:21.496745 26938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:21.497431 28465 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:21.497560 28465 ts_tablet_manager.cc:1367] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:21.498298 28466 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:21.498605 28466 ts_tablet_manager.cc:1367] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 20:46:21.498670 28465 raft_consensus.cc:348] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.498848 28465 raft_consensus.cc:374] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:21.498898 28465 raft_consensus.cc:729] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:21.499089 28465 consensus_queue.cc:260] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.499480 28465 ts_tablet_manager.cc:1398] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:21.499540 28466 raft_consensus.cc:348] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.499706 28466 raft_consensus.cc:374] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:21.499738 28465 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:21.499751 28466 raft_consensus.cc:729] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:21.499907 28466 consensus_queue.cc:260] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.500641 28466 ts_tablet_manager.cc:1398] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 20:46:21.500979 28465 tablet_bootstrap.cc:654] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:21.503278 28465 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:21.503446 28465 ts_tablet_manager.cc:1367] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 20:46:21.504299 28465 raft_consensus.cc:348] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.504642 28465 raft_consensus.cc:374] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:21.504765 28465 raft_consensus.cc:729] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:21.504963 28465 consensus_queue.cc:260] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.505434 28465 ts_tablet_manager.cc:1398] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 20:46:21.508467 28470 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:21.510354 28470 tablet_bootstrap.cc:654] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:21.512085 28470 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:21.512221 28470 ts_tablet_manager.cc:1367] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:46:21.512706 28470 raft_consensus.cc:348] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.512826 28470 raft_consensus.cc:374] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:21.512864 28470 raft_consensus.cc:729] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:21.513029 28470 consensus_queue.cc:260] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.513998 28470 ts_tablet_manager.cc:1398] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:21.515112 28470 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:21.516373 28470 tablet_bootstrap.cc:654] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:21.516669 28472 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:21.517760 28472 tablet_bootstrap.cc:654] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:21.519511 28470 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:21.519502 28472 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:21.519630 28472 ts_tablet_manager.cc:1367] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:21.519748 28470 ts_tablet_manager.cc:1367] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 20:46:21.520172 28472 raft_consensus.cc:348] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.520324 28472 raft_consensus.cc:374] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:21.520375 28472 raft_consensus.cc:729] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:21.520300 28470 raft_consensus.cc:348] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.520429 28470 raft_consensus.cc:374] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:21.520473 28470 raft_consensus.cc:729] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:21.520532 28472 consensus_queue.cc:260] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.520599 28470 consensus_queue.cc:260] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.520934 28470 ts_tablet_manager.cc:1398] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:21.520934 28472 ts_tablet_manager.cc:1398] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:21.521174 28470 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:21.521272 28472 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:21.522361 28470 tablet_bootstrap.cc:654] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:21.523607 28472 tablet_bootstrap.cc:654] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:21.524252 28470 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:21.524363 28470 ts_tablet_manager.cc:1367] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:21.524960 28470 raft_consensus.cc:348] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.525146 28470 raft_consensus.cc:374] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:21.525192 28470 raft_consensus.cc:729] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:21.525372 28470 consensus_queue.cc:260] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.525839 28470 ts_tablet_manager.cc:1398] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:21.526450 28472 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:21.526566 28472 ts_tablet_manager.cc:1367] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:46:21.526981 28472 raft_consensus.cc:348] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.527102 28472 raft_consensus.cc:374] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:21.527137 28472 raft_consensus.cc:729] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:21.527294 28472 consensus_queue.cc:260] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.527571 28472 ts_tablet_manager.cc:1398] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:21.527714 28472 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:21.529290 28472 tablet_bootstrap.cc:654] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:21.530647 28472 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:21.530778 28472 ts_tablet_manager.cc:1367] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:21.531296 28472 raft_consensus.cc:348] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.531410 28472 raft_consensus.cc:374] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:21.531435 28472 raft_consensus.cc:729] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:21.531567 28472 consensus_queue.cc:260] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.532024 28472 ts_tablet_manager.cc:1398] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:21.676599 28461 raft_consensus.cc:482] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:21.676779 28461 raft_consensus.cc:504] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.677366 28461 leader_election.cc:288] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:21.677958 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f81e46ab1f48fc8a88495b42f83640" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:21.677958 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f81e46ab1f48fc8a88495b42f83640" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:21.678110 26959 raft_consensus.cc:2454] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:21.678133 27362 raft_consensus.cc:2454] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:21.678510 27158 leader_election.cc:302] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:21.678748 28461 raft_consensus.cc:2781] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:21.678822 28461 raft_consensus.cc:482] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:21.678854 28461 raft_consensus.cc:3037] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.679852 28461 raft_consensus.cc:504] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.680248 28461 leader_election.cc:288] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Requested vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:21.680775 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f81e46ab1f48fc8a88495b42f83640" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:21.680774 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5f81e46ab1f48fc8a88495b42f83640" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:21.680928 26959 raft_consensus.cc:3037] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.680928 27362 raft_consensus.cc:3037] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.682063 27362 raft_consensus.cc:2454] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:21.682063 26959 raft_consensus.cc:2454] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:21.682435 27160 leader_election.cc:302] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:21.682644 28461 raft_consensus.cc:2781] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:21.682746 28461 raft_consensus.cc:686] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Leader. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:21.682917 28461 consensus_queue.cc:237] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.685221 26807 catalog_manager.cc:5484] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:21.709704 28095 raft_consensus.cc:482] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:21.709895 28095 raft_consensus.cc:504] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.710407 28095 leader_election.cc:288] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:21.711112 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5704bdfe6c02437c8e9c8c3deaadb9e3" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:21.711112 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5704bdfe6c02437c8e9c8c3deaadb9e3" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:21.711288 27362 raft_consensus.cc:2454] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:21.711288 26959 raft_consensus.cc:2454] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:21.711683 27026 leader_election.cc:302] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:21.711865 28095 raft_consensus.cc:2781] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:21.711928 28095 raft_consensus.cc:482] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:21.711968 28095 raft_consensus.cc:3037] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.712919 28095 raft_consensus.cc:504] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.713338 28095 leader_election.cc:288] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Requested vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:21.713851 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5704bdfe6c02437c8e9c8c3deaadb9e3" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:21.713851 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5704bdfe6c02437c8e9c8c3deaadb9e3" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:21.713996 27362 raft_consensus.cc:3037] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.713996 26959 raft_consensus.cc:3037] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.715076 26959 raft_consensus.cc:2454] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:21.715076 27362 raft_consensus.cc:2454] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:21.715445 27026 leader_election.cc:302] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:21.715687 28095 raft_consensus.cc:2781] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:21.715759 28095 raft_consensus.cc:686] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Becoming Leader. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:21.715914 28095 consensus_queue.cc:237] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:21.718782 26807 catalog_manager.cc:5484] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:21.904183 28095 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:21.904433 28095 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.905160 28095 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:21.905992 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:21.906024 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:21.906212 27362 raft_consensus.cc:2454] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:21.906242 26959 raft_consensus.cc:2454] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:21.906687 27024 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:21.906965 28095 raft_consensus.cc:2781] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:21.907044 28095 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:21.907102 28095 raft_consensus.cc:3037] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.908264 28095 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.908694 28095 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Requested vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:21.909308 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:21.909368 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:21.909503 27362 raft_consensus.cc:3037] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.909524 26959 raft_consensus.cc:3037] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:21.910674 26959 raft_consensus.cc:2454] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:21.910674 27362 raft_consensus.cc:2454] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:21.911124 27026 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:21.911403 28095 raft_consensus.cc:2781] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:21.911530 28095 raft_consensus.cc:686] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Becoming Leader. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:21.911701 28095 consensus_queue.cc:237] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:21.915089 26807 catalog_manager.cc:5484] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:22.107745 28444 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:46:22.113104 28481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:22.113550 28481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 20:46:22.116111 28461 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 20:46:22.119581 28481 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:46:22.127738 28095 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:22.133421 28407 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240502 20:46:22.255338 26957 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:22.256054 28095 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:22.257418 27361 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:22.259828 28095 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 136.274 ms
  time per row: 0.068137 ms
I20240502 20:46:22.333761 26802 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47424:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:46:22.350895 27207 tablet_service.cc:1467] Processing CreateTablet for tablet 135a08f1f2ac4707bbacee81ac73a19e (DEFAULT_TABLE table=rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:22.350931 27070 tablet_service.cc:1467] Processing CreateTablet for tablet 135a08f1f2ac4707bbacee81ac73a19e (DEFAULT_TABLE table=rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:22.351372 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135a08f1f2ac4707bbacee81ac73a19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:22.351408 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135a08f1f2ac4707bbacee81ac73a19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:22.352115 27340 tablet_service.cc:1467] Processing CreateTablet for tablet cd1b1316da2d47938164ac5b75f11bae (DEFAULT_TABLE table=rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:22.352484 27340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1b1316da2d47938164ac5b75f11bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:22.354986 28541 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:22.355844 27342 tablet_service.cc:1467] Processing CreateTablet for tablet a547561343304137a044ce11517f6007 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:22.356187 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a547561343304137a044ce11517f6007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:22.356324 28541 tablet_bootstrap.cc:654] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:22.356853 28542 tablet_bootstrap.cc:492] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:22.358755 27070 tablet_service.cc:1467] Processing CreateTablet for tablet a547561343304137a044ce11517f6007 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:22.358835 27204 tablet_service.cc:1467] Processing CreateTablet for tablet a547561343304137a044ce11517f6007 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:22.359066 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a547561343304137a044ce11517f6007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:22.359148 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a547561343304137a044ce11517f6007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:22.360994 27207 tablet_service.cc:1467] Processing CreateTablet for tablet cd1b1316da2d47938164ac5b75f11bae (DEFAULT_TABLE table=rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:22.361367 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1b1316da2d47938164ac5b75f11bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:22.362766 28540 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:22.363548 28542 tablet_bootstrap.cc:654] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:22.364002 28540 tablet_bootstrap.cc:654] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:22.364338 26938 tablet_service.cc:1467] Processing CreateTablet for tablet 135a08f1f2ac4707bbacee81ac73a19e (DEFAULT_TABLE table=rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:22.364637 26938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135a08f1f2ac4707bbacee81ac73a19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:22.365526 28541 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:22.365551 28540 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:22.365641 28541 ts_tablet_manager.cc:1367] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.000s
I20240502 20:46:22.365674 28540 ts_tablet_manager.cc:1367] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:22.366223 28541 raft_consensus.cc:348] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.366350 28541 raft_consensus.cc:374] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:22.366255 28540 raft_consensus.cc:348] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.366384 28541 raft_consensus.cc:729] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:22.366391 28540 raft_consensus.cc:374] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:22.366427 28540 raft_consensus.cc:729] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:22.366539 28541 consensus_queue.cc:260] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.366830 28541 ts_tablet_manager.cc:1398] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:22.366869 28540 consensus_queue.cc:260] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.367003 28541 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:22.367233 28540 ts_tablet_manager.cc:1398] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:46:22.367738 26936 tablet_service.cc:1467] Processing CreateTablet for tablet cd1b1316da2d47938164ac5b75f11bae (DEFAULT_TABLE table=rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:22.368041 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1b1316da2d47938164ac5b75f11bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:22.368059 28541 tablet_bootstrap.cc:654] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:22.368145 28540 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:22.369191 28540 tablet_bootstrap.cc:654] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:22.370090 28541 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:22.370220 28541 ts_tablet_manager.cc:1367] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:46:22.370828 28541 raft_consensus.cc:348] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:22.370977 28541 raft_consensus.cc:374] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:22.371009 28541 raft_consensus.cc:729] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:22.371138 28541 consensus_queue.cc:260] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:22.371569 28541 ts_tablet_manager.cc:1398] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:22.374082 28540 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:22.374078 28542 tablet_bootstrap.cc:492] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:22.374223 28540 ts_tablet_manager.cc:1367] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.002s
I20240502 20:46:22.374223 28542 ts_tablet_manager.cc:1367] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.017s	user 0.003s	sys 0.000s
I20240502 20:46:22.374570 28546 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:22.374835 28540 raft_consensus.cc:348] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:22.374833 28542 raft_consensus.cc:348] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.374977 28542 raft_consensus.cc:374] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:22.374977 28540 raft_consensus.cc:374] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:22.375013 28540 raft_consensus.cc:729] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:22.375013 28542 raft_consensus.cc:729] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:22.375170 28540 consensus_queue.cc:260] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:22.375169 28542 consensus_queue.cc:260] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.375517 28542 ts_tablet_manager.cc:1398] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:22.375545 28540 ts_tablet_manager.cc:1398] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:22.375715 28542 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:22.375723 28540 tablet_bootstrap.cc:492] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:22.375978 28546 tablet_bootstrap.cc:654] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:22.376945 28542 tablet_bootstrap.cc:654] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:22.376977 28540 tablet_bootstrap.cc:654] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:22.377565 28546 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:22.377683 28546 ts_tablet_manager.cc:1367] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:46:22.379129 28542 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:22.379248 28542 ts_tablet_manager.cc:1367] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:46:22.379812 28542 raft_consensus.cc:348] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:22.379958 28542 raft_consensus.cc:374] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:22.379993 28542 raft_consensus.cc:729] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:22.380154 28542 consensus_queue.cc:260] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:22.380509 28542 ts_tablet_manager.cc:1398] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:22.381453 28540 tablet_bootstrap.cc:492] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:22.381569 28540 ts_tablet_manager.cc:1367] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:46:22.381884 28546 raft_consensus.cc:348] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.382129 28546 raft_consensus.cc:374] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:22.382200 28546 raft_consensus.cc:729] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:22.382129 28540 raft_consensus.cc:348] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.382258 28540 raft_consensus.cc:374] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:22.382289 28540 raft_consensus.cc:729] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:22.382412 28540 consensus_queue.cc:260] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.382391 28546 consensus_queue.cc:260] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.382728 28540 ts_tablet_manager.cc:1398] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:22.382756 28546 ts_tablet_manager.cc:1398] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.000s
I20240502 20:46:22.382992 28546 tablet_bootstrap.cc:492] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:22.384495 28546 tablet_bootstrap.cc:654] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:22.386101 28546 tablet_bootstrap.cc:492] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:22.386204 28546 ts_tablet_manager.cc:1367] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:22.386778 28546 raft_consensus.cc:348] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.386924 28546 raft_consensus.cc:374] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:22.386977 28546 raft_consensus.cc:729] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:22.387132 28546 consensus_queue.cc:260] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.387475 28546 ts_tablet_manager.cc:1398] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:22.497812 28407 raft_consensus.cc:482] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:22.498028 28407 raft_consensus.cc:504] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.498667 28407 leader_election.cc:288] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:22.499241 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135a08f1f2ac4707bbacee81ac73a19e" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:22.499276 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135a08f1f2ac4707bbacee81ac73a19e" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:22.499444 26959 raft_consensus.cc:2454] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:22.499568 27228 raft_consensus.cc:2454] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:22.499936 27024 leader_election.cc:302] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:22.500202 28407 raft_consensus.cc:2781] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:22.500298 28407 raft_consensus.cc:482] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:22.500331 28407 raft_consensus.cc:3037] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:22.501916 28407 raft_consensus.cc:504] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.502446 28407 leader_election.cc:288] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Requested vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:22.503113 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135a08f1f2ac4707bbacee81ac73a19e" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:22.503113 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135a08f1f2ac4707bbacee81ac73a19e" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:22.503342 26959 raft_consensus.cc:3037] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:22.503345 27228 raft_consensus.cc:3037] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:22.504604 27228 raft_consensus.cc:2454] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:22.504604 26959 raft_consensus.cc:2454] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:22.505198 27024 leader_election.cc:302] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:22.505620 28407 raft_consensus.cc:2781] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:22.505731 28407 raft_consensus.cc:686] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Becoming Leader. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:22.505914 28407 consensus_queue.cc:237] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.508316 26807 catalog_manager.cc:5484] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:22.589499 28444 raft_consensus.cc:482] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:22.589767 28444 raft_consensus.cc:504] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.590454 28444 leader_election.cc:288] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:22.591068 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1b1316da2d47938164ac5b75f11bae" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:22.591367 26959 raft_consensus.cc:2454] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:22.591522 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1b1316da2d47938164ac5b75f11bae" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:22.591789 27361 raft_consensus.cc:2454] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:22.591848 27158 leader_election.cc:302] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:22.592125 28444 raft_consensus.cc:2781] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:22.592234 28444 raft_consensus.cc:482] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:22.592271 28444 raft_consensus.cc:3037] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:22.593639 28444 raft_consensus.cc:504] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.594187 28444 leader_election.cc:288] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Requested vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:22.594944 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1b1316da2d47938164ac5b75f11bae" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:22.594944 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1b1316da2d47938164ac5b75f11bae" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:22.595160 26959 raft_consensus.cc:3037] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:22.595193 27361 raft_consensus.cc:3037] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:22.596606 26959 raft_consensus.cc:2454] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:22.596606 27361 raft_consensus.cc:2454] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:22.597143 27158 leader_election.cc:302] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:22.597397 28444 raft_consensus.cc:2781] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:22.597512 28444 raft_consensus.cc:686] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Leader. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:22.597703 28444 consensus_queue.cc:237] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:22.600291 26807 catalog_manager.cc:5484] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:22.677326 28444 raft_consensus.cc:482] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:22.677520 28444 raft_consensus.cc:504] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:22.678113 28444 leader_election.cc:288] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:22.678851 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a547561343304137a044ce11517f6007" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:22.678850 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a547561343304137a044ce11517f6007" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:22.679036 27361 raft_consensus.cc:2454] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:22.679080 27092 raft_consensus.cc:2454] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:22.679541 27160 leader_election.cc:302] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:22.679814 28444 raft_consensus.cc:2781] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:22.679893 28444 raft_consensus.cc:482] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:22.679937 28444 raft_consensus.cc:3037] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:22.681164 28444 raft_consensus.cc:504] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:22.681593 28444 leader_election.cc:288] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Requested vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:22.682194 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a547561343304137a044ce11517f6007" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:22.682297 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a547561343304137a044ce11517f6007" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:22.682384 27361 raft_consensus.cc:3037] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:22.682497 27092 raft_consensus.cc:3037] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:22.683513 27361 raft_consensus.cc:2454] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:22.683667 27092 raft_consensus.cc:2454] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:22.684101 27160 leader_election.cc:302] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:22.684420 28444 raft_consensus.cc:2781] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:22.684551 28444 raft_consensus.cc:686] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Leader. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:22.684832 28444 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:22.687664 26807 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:22.891952 28557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:22.892634 28557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:22.897568 28557 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 20:46:23.010257 26959 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:23.010341 27228 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:23.011066 28407 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.011277 28095 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.013695 27361 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:23.014277 28461 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.016865 27092 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:23.017983 28461 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.022397 27361 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:23.022513 26960 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:23.023025 28444 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.023272 28461 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.039920 28590 mvcc.cc:205] Tried to move back new op lower bound from 7023340679204036608 to 7023340677146087424. Current Snapshot: MvccSnapshot[applied={T|T < 7023340679204036608}]

INSERT report
    rows total: 2000
    time total: 140.403 ms
  time per row: 0.0702013 ms
I20240502 20:46:23.111673 26806 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47424:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:46:23.122083 26936 tablet_service.cc:1467] Processing CreateTablet for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.122076 27340 tablet_service.cc:1467] Processing CreateTablet for tablet 40df15a82a624913b3b712687d999134 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.122550 27340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40df15a82a624913b3b712687d999134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.122548 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.122076 27342 tablet_service.cc:1467] Processing CreateTablet for tablet f00d12e425284c36ae8baed56c7cfaf6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.122967 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00d12e425284c36ae8baed56c7cfaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.125607 26938 tablet_service.cc:1467] Processing CreateTablet for tablet 40df15a82a624913b3b712687d999134 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.125963 26938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40df15a82a624913b3b712687d999134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.128062 28617 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:23.128716 27204 tablet_service.cc:1467] Processing CreateTablet for tablet f00d12e425284c36ae8baed56c7cfaf6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.129038 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00d12e425284c36ae8baed56c7cfaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.129328 28617 tablet_bootstrap.cc:654] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.129384 27207 tablet_service.cc:1467] Processing CreateTablet for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.129640 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.130501 28617 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:23.130576 28617 ts_tablet_manager.cc:1367] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:46:23.130977 28617 raft_consensus.cc:348] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.131067 28617 raft_consensus.cc:374] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.131088 28617 raft_consensus.cc:729] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.131175 28617 consensus_queue.cc:260] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.132007 28619 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:23.132396 26938 tablet_service.cc:1467] Processing CreateTablet for tablet f00d12e425284c36ae8baed56c7cfaf6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.132735 26938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00d12e425284c36ae8baed56c7cfaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.133291 27070 tablet_service.cc:1467] Processing CreateTablet for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.133620 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.135129 28619 tablet_bootstrap.cc:654] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.135124 28617 ts_tablet_manager.cc:1398] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.001s
I20240502 20:46:23.135711 28617 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:23.136579 28619 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:23.136700 28619 ts_tablet_manager.cc:1367] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:46:23.136850 27071 tablet_service.cc:1467] Processing CreateTablet for tablet 40df15a82a624913b3b712687d999134 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.137213 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40df15a82a624913b3b712687d999134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.137333 28619 raft_consensus.cc:348] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.137491 28619 raft_consensus.cc:374] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.137530 28619 raft_consensus.cc:729] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.137997 28616 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:23.137941 28619 consensus_queue.cc:260] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.138615 28619 ts_tablet_manager.cc:1398] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:23.138900 28619 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:23.139626 28617 tablet_bootstrap.cc:654] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.139873 28616 tablet_bootstrap.cc:654] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.140053 28619 tablet_bootstrap.cc:654] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.141206 28617 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:23.141336 28617 ts_tablet_manager.cc:1367] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.002s
I20240502 20:46:23.141527 28619 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:23.141625 28619 ts_tablet_manager.cc:1367] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:23.141757 28616 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:23.141949 28617 raft_consensus.cc:348] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.142102 28617 raft_consensus.cc:374] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.142140 28617 raft_consensus.cc:729] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.142159 28621 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:23.142310 28617 consensus_queue.cc:260] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.142418 28619 raft_consensus.cc:348] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.142567 28619 raft_consensus.cc:374] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.142608 28619 raft_consensus.cc:729] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.142683 28617 ts_tablet_manager.cc:1398] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:23.142782 28619 consensus_queue.cc:260] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.142920 28617 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:23.143178 28619 ts_tablet_manager.cc:1398] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
I20240502 20:46:23.143687 28616 ts_tablet_manager.cc:1367] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 20:46:23.144472 28621 tablet_bootstrap.cc:654] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.145201 28616 raft_consensus.cc:348] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.145395 28616 raft_consensus.cc:374] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.145440 28616 raft_consensus.cc:729] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.145625 28616 consensus_queue.cc:260] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.145701 28617 tablet_bootstrap.cc:654] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.146214 28621 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:23.146348 28621 ts_tablet_manager.cc:1367] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 20:46:23.146785 28616 ts_tablet_manager.cc:1398] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:46:23.146932 28621 raft_consensus.cc:348] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.147078 28621 raft_consensus.cc:374] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.147127 28621 raft_consensus.cc:729] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.147285 28621 consensus_queue.cc:260] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.147639 28621 ts_tablet_manager.cc:1398] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:23.147642 28616 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:23.147837 28621 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:23.148762 28621 tablet_bootstrap.cc:654] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.151007 28621 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:23.151111 28621 ts_tablet_manager.cc:1367] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 20:46:23.151149 28616 tablet_bootstrap.cc:654] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.151577 28621 raft_consensus.cc:348] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.151721 28621 raft_consensus.cc:374] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.151760 28621 raft_consensus.cc:729] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.151891 28621 consensus_queue.cc:260] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.152485 28621 ts_tablet_manager.cc:1398] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:23.153190 28616 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:23.153537 28616 ts_tablet_manager.cc:1367] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 20:46:23.153630 28617 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:23.153740 28617 ts_tablet_manager.cc:1367] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.001s
I20240502 20:46:23.154079 28616 raft_consensus.cc:348] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.154194 28616 raft_consensus.cc:374] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.154218 28616 raft_consensus.cc:729] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.154261 28617 raft_consensus.cc:348] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.154387 28617 raft_consensus.cc:374] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.154434 28617 raft_consensus.cc:729] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.154385 28616 consensus_queue.cc:260] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.154585 28617 consensus_queue.cc:260] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.154803 28616 ts_tablet_manager.cc:1398] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:23.154860 28617 ts_tablet_manager.cc:1398] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:23.155876 28439 raft_consensus.cc:482] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:23.155990 28439 raft_consensus.cc:504] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.156399 28439 leader_election.cc:288] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:23.157362 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40df15a82a624913b3b712687d999134" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:23.157595 27092 raft_consensus.cc:2454] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:23.157979 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40df15a82a624913b3b712687d999134" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:23.158166 26959 raft_consensus.cc:2454] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:23.158540 27298 leader_election.cc:302] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:23.158787 28439 raft_consensus.cc:2781] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:23.158933 28439 raft_consensus.cc:482] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:23.158991 28439 raft_consensus.cc:3037] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.160842 28439 raft_consensus.cc:504] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.161342 28439 leader_election.cc:288] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Requested vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:23.161814 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40df15a82a624913b3b712687d999134" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:23.161816 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40df15a82a624913b3b712687d999134" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:23.161943 27092 raft_consensus.cc:3037] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.161954 26959 raft_consensus.cc:3037] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.162910 27092 raft_consensus.cc:2454] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:23.163360 27298 leader_election.cc:302] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:23.163422 26959 raft_consensus.cc:2454] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:23.163658 28439 raft_consensus.cc:2781] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:23.163892 28439 raft_consensus.cc:686] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Becoming Leader. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Running, Role: LEADER
I20240502 20:46:23.164091 28439 consensus_queue.cc:237] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.167268 26806 catalog_manager.cc:5484] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132). New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:23.354934 28439 raft_consensus.cc:482] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:23.355146 28439 raft_consensus.cc:504] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.355916 28439 leader_election.cc:288] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:23.356844 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00d12e425284c36ae8baed56c7cfaf6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:23.356806 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00d12e425284c36ae8baed56c7cfaf6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:23.357100 27228 raft_consensus.cc:2454] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:23.357154 26959 raft_consensus.cc:2454] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:23.357570 27295 leader_election.cc:302] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:23.357847 28439 raft_consensus.cc:2781] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:23.357933 28439 raft_consensus.cc:482] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:23.357975 28439 raft_consensus.cc:3037] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.359475 28439 raft_consensus.cc:504] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.360002 28439 leader_election.cc:288] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Requested vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:23.360489 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00d12e425284c36ae8baed56c7cfaf6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:23.360538 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00d12e425284c36ae8baed56c7cfaf6" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:23.360666 26959 raft_consensus.cc:3037] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.360661 27228 raft_consensus.cc:3037] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.361840 27228 raft_consensus.cc:2454] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:23.361840 26959 raft_consensus.cc:2454] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:23.362263 27298 leader_election.cc:302] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:23.362509 28439 raft_consensus.cc:2781] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:23.362638 28439 raft_consensus.cc:686] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Becoming Leader. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Running, Role: LEADER
I20240502 20:46:23.362622 28407 raft_consensus.cc:482] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:23.362728 28407 raft_consensus.cc:504] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.362811 28439 consensus_queue.cc:237] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:46:23.363619 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ebc6576a4e4a108de55a4e9d7ff6b8" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:23.363755 27228 raft_consensus.cc:2454] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:23.364092 27027 leader_election.cc:302] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:23.364226 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ebc6576a4e4a108de55a4e9d7ff6b8" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:23.364274 28095 raft_consensus.cc:2781] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:23.364321 28095 raft_consensus.cc:482] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:23.364377 28095 raft_consensus.cc:3037] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.364373 26959 raft_consensus.cc:2454] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 0.
I20240502 20:46:23.365738 28095 raft_consensus.cc:504] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.365744 28407 leader_election.cc:288] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:23.366472 28095 leader_election.cc:288] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Requested vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:23.366904 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ebc6576a4e4a108de55a4e9d7ff6b8" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:23.366907 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ebc6576a4e4a108de55a4e9d7ff6b8" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3"
I20240502 20:46:23.367076 26959 raft_consensus.cc:3037] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.367075 27228 raft_consensus.cc:3037] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.367300 26806 catalog_manager.cc:5484] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132). New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:23.368301 27228 raft_consensus.cc:2454] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:23.368834 27027 leader_election.cc:302] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:23.368965 26959 raft_consensus.cc:2454] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 1.
I20240502 20:46:23.369494 28095 raft_consensus.cc:2781] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:23.369618 28095 raft_consensus.cc:686] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Becoming Leader. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:23.369789 28095 consensus_queue.cc:237] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.372834 26806 catalog_manager.cc:5484] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130). New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:23.535858 28632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:23.536254 28632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:23.540091 28632 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:46:23.565785 28439 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.571597 28439 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 20:46:23.672287 26960 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:23.672933 28644 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.674679 27228 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:23.685520 28439 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.688688 26957 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:23.689293 28095 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.689482 27228 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:23.690304 28407 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:23.741209 28591 mvcc.cc:205] Tried to move back new op lower bound from 7023340681981120512 to 7023340680684613632. Current Snapshot: MvccSnapshot[applied={T|T < 7023340681981120512}]
I20240502 20:46:23.742161 28678 mvcc.cc:205] Tried to move back new op lower bound from 7023340681902813184 to 7023340680655601664. Current Snapshot: MvccSnapshot[applied={T|T < 7023340681902813184}]

INSERT report
    rows total: 2000
    time total: 167.808 ms
  time per row: 0.0839038 ms
I20240502 20:46:23.788501 26806 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47424:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 20:46:23.797037 27071 tablet_service.cc:1467] Processing CreateTablet for tablet f1545142b0bd40859bf33bf8686002ae (DEFAULT_TABLE table=rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.797216 27342 tablet_service.cc:1467] Processing CreateTablet for tablet f1545142b0bd40859bf33bf8686002ae (DEFAULT_TABLE table=rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.797305 27204 tablet_service.cc:1467] Processing CreateTablet for tablet f1545142b0bd40859bf33bf8686002ae (DEFAULT_TABLE table=rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.797430 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1545142b0bd40859bf33bf8686002ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.797571 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1545142b0bd40859bf33bf8686002ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.797571 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1545142b0bd40859bf33bf8686002ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.799618 28692 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:23.800918 28693 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:23.801010 28692 tablet_bootstrap.cc:654] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.801662 27071 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf737e20ca74db1b94ccff5d60133ee (DEFAULT_TABLE table=rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.801963 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf737e20ca74db1b94ccff5d60133ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.802054 28693 tablet_bootstrap.cc:654] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.802445 26936 tablet_service.cc:1467] Processing CreateTablet for tablet af693f5816e046f592fbd9b2e7ef0ad9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.802733 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693f5816e046f592fbd9b2e7ef0ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.803079 28694 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:23.804316 28694 tablet_bootstrap.cc:654] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.804423 27204 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf737e20ca74db1b94ccff5d60133ee (DEFAULT_TABLE table=rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.804744 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf737e20ca74db1b94ccff5d60133ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.806030 28693 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:23.806301 28693 ts_tablet_manager.cc:1367] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 20:46:23.806587 27207 tablet_service.cc:1467] Processing CreateTablet for tablet af693f5816e046f592fbd9b2e7ef0ad9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.807163 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693f5816e046f592fbd9b2e7ef0ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.807225 27071 tablet_service.cc:1467] Processing CreateTablet for tablet af693f5816e046f592fbd9b2e7ef0ad9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.807551 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693f5816e046f592fbd9b2e7ef0ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.807569 28693 raft_consensus.cc:348] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.807741 28693 raft_consensus.cc:374] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.807781 28693 raft_consensus.cc:729] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.807936 28693 consensus_queue.cc:260] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.808451 28693 ts_tablet_manager.cc:1398] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:46:23.806304 27342 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf737e20ca74db1b94ccff5d60133ee (DEFAULT_TABLE table=rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 20:46:23.809733 27342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf737e20ca74db1b94ccff5d60133ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:23.811698 28692 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:23.811839 28693 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:23.811836 28692 ts_tablet_manager.cc:1367] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.000s
I20240502 20:46:23.812460 28692 raft_consensus.cc:348] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.812623 28692 raft_consensus.cc:374] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.812659 28692 raft_consensus.cc:729] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.812822 28692 consensus_queue.cc:260] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.813045 28693 tablet_bootstrap.cc:654] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.813216 28692 ts_tablet_manager.cc:1398] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:23.813431 28692 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:23.814416 28692 tablet_bootstrap.cc:654] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.814858 28693 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: No bootstrap required, opened a new log
I20240502 20:46:23.814982 28693 ts_tablet_manager.cc:1367] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:46:23.815554 28693 raft_consensus.cc:348] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:23.815711 28693 raft_consensus.cc:374] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.815748 28693 raft_consensus.cc:729] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.815892 28693 consensus_queue.cc:260] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:23.816370 28693 ts_tablet_manager.cc:1398] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:23.818032 28694 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:23.818145 28694 ts_tablet_manager.cc:1367] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.015s	user 0.000s	sys 0.002s
I20240502 20:46:23.818377 28692 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:23.818465 28692 ts_tablet_manager.cc:1367] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:46:23.818701 28694 raft_consensus.cc:348] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.818818 28694 raft_consensus.cc:374] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.818969 28692 raft_consensus.cc:348] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:23.819089 28692 raft_consensus.cc:374] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.819131 28692 raft_consensus.cc:729] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.819260 28692 consensus_queue.cc:260] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:23.819620 28692 ts_tablet_manager.cc:1398] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:23.819819 28692 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:23.818853 28694 raft_consensus.cc:729] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.820447 28694 consensus_queue.cc:260] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.820895 28694 ts_tablet_manager.cc:1398] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240502 20:46:23.821182 28694 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:23.821354 28692 tablet_bootstrap.cc:654] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.822022 28699 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:46:23.822477 28694 tablet_bootstrap.cc:654] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.823242 28699 tablet_bootstrap.cc:654] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.824839 28699 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: No bootstrap required, opened a new log
I20240502 20:46:23.824959 28699 ts_tablet_manager.cc:1367] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.000s
I20240502 20:46:23.825389 28694 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:23.825481 28694 ts_tablet_manager.cc:1367] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 20:46:23.825978 28692 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997: No bootstrap required, opened a new log
I20240502 20:46:23.825974 28694 raft_consensus.cc:348] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:23.826071 28692 ts_tablet_manager.cc:1367] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:46:23.826090 28694 raft_consensus.cc:374] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.826125 28694 raft_consensus.cc:729] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.826254 28694 consensus_queue.cc:260] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:23.826454 28692 raft_consensus.cc:348] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:23.826539 28692 raft_consensus.cc:374] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.826536 28694 ts_tablet_manager.cc:1398] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:23.826560 28692 raft_consensus.cc:729] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.826714 28694 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:46:23.826705 28692 consensus_queue.cc:260] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:23.826961 28692 ts_tablet_manager.cc:1398] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:23.827100 28699 raft_consensus.cc:348] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:23.827220 28699 raft_consensus.cc:374] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.827252 28699 raft_consensus.cc:729] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.827395 28699 consensus_queue.cc:260] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:23.827593 28694 tablet_bootstrap.cc:654] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Neither blocks nor log segments found. Creating new log.
I20240502 20:46:23.827661 28699 ts_tablet_manager.cc:1398] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:46:23.828763 28694 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: No bootstrap required, opened a new log
I20240502 20:46:23.828859 28694 ts_tablet_manager.cc:1367] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 20:46:23.829393 28694 raft_consensus.cc:348] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:23.829524 28694 raft_consensus.cc:374] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:46:23.829578 28694 raft_consensus.cc:729] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: FOLLOWER
I20240502 20:46:23.829699 28694 consensus_queue.cc:260] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:23.829990 28694 ts_tablet_manager.cc:1398] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:23.936967 28612 raft_consensus.cc:482] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:23.937276 28612 raft_consensus.cc:504] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.937898 28612 leader_election.cc:288] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:23.938669 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1545142b0bd40859bf33bf8686002ae" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:23.938751 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1545142b0bd40859bf33bf8686002ae" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:23.938856 27362 raft_consensus.cc:2454] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:23.939021 27093 raft_consensus.cc:2454] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 0.
I20240502 20:46:23.939433 27160 leader_election.cc:302] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:23.939774 28612 raft_consensus.cc:2781] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:23.939865 28612 raft_consensus.cc:482] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:23.939909 28612 raft_consensus.cc:3037] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.941124 28612 raft_consensus.cc:504] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.941605 28612 leader_election.cc:288] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Requested vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:23.941601 28527 raft_consensus.cc:482] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:23.941745 28527 raft_consensus.cc:504] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:23.942001 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1545142b0bd40859bf33bf8686002ae" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:23.942065 27362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1545142b0bd40859bf33bf8686002ae" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:23.942186 27362 raft_consensus.cc:3037] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.942188 27093 raft_consensus.cc:3037] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.942248 28527 leader_election.cc:288] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:23.942693 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf737e20ca74db1b94ccff5d60133ee" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:23.942839 27092 raft_consensus.cc:2454] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:23.943171 27298 leader_election.cc:302] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:23.943356 28527 raft_consensus.cc:2781] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:23.943363 27093 raft_consensus.cc:2454] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:23.943449 28527 raft_consensus.cc:482] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:23.943362 27362 raft_consensus.cc:2454] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c107c43e538d48f58625c4320f0aa4b8 in term 1.
I20240502 20:46:23.943493 28527 raft_consensus.cc:3037] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.943753 27161 leader_election.cc:302] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 
I20240502 20:46:23.943763 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf737e20ca74db1b94ccff5d60133ee" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:23.943914 27228 raft_consensus.cc:2454] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 0.
I20240502 20:46:23.944041 28612 raft_consensus.cc:2781] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:23.944439 28612 raft_consensus.cc:686] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Leader. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:23.944425 28646 raft_consensus.cc:482] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:23.944559 28646 raft_consensus.cc:504] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:23.944631 28612 consensus_queue.cc:237] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } }
I20240502 20:46:23.944876 28527 raft_consensus.cc:504] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:23.945158 28646 leader_election.cc:288] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:23.945312 28527 leader_election.cc:288] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Requested vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:23.945998 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf737e20ca74db1b94ccff5d60133ee" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:23.946143 27228 raft_consensus.cc:3037] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.946336 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af693f5816e046f592fbd9b2e7ef0ad9" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:23.946478 27227 raft_consensus.cc:2454] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 0.
I20240502 20:46:23.946697 27092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af693f5816e046f592fbd9b2e7ef0ad9" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:23.946866 26894 leader_election.cc:302] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:23.946902 27092 raft_consensus.cc:2454] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 0.
I20240502 20:46:23.947086 28646 raft_consensus.cc:2781] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 20:46:23.947139 28646 raft_consensus.cc:482] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 20:46:23.947175 28646 raft_consensus.cc:3037] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.946718 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf737e20ca74db1b94ccff5d60133ee" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:23.947257 27093 raft_consensus.cc:3037] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.947718 27228 raft_consensus.cc:2454] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:23.948186 27298 leader_election.cc:302] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:23.948400 27093 raft_consensus.cc:2454] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 1.
I20240502 20:46:23.948482 28527 raft_consensus.cc:2781] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:23.948508 28646 raft_consensus.cc:504] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:23.948607 26805 catalog_manager.cc:5484] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131). New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } }
I20240502 20:46:23.948967 28646 leader_election.cc:288] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 election: Requested vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:23.948971 28527 raft_consensus.cc:686] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Becoming Leader. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Running, Role: LEADER
I20240502 20:46:23.949303 28527 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } }
I20240502 20:46:23.949391 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af693f5816e046f592fbd9b2e7ef0ad9" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:23.949450 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af693f5816e046f592fbd9b2e7ef0ad9" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:23.949493 27228 raft_consensus.cc:3037] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.949568 27093 raft_consensus.cc:3037] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:46:23.950650 27228 raft_consensus.cc:2454] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 1.
I20240502 20:46:23.951006 26894 leader_election.cc:302] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:23.951208 28646 raft_consensus.cc:2781] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:46:23.951303 28646 raft_consensus.cc:686] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Becoming Leader. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Running, Role: LEADER
I20240502 20:46:23.951465 28646 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:46:23.951782 27093 raft_consensus.cc:2454] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 1.
I20240502 20:46:23.953908 26805 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 297de823852d4a8da44791ab03131cd3 (127.19.66.129). New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:23.954886 26806 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132). New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } }
W20240502 20:46:24.126021 28709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:24.126579 28709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:24.130609 28709 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 20:46:24.250824 27362 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:24.250825 27093 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:24.251416 28444 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:24.251547 27093 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:24.252146 28646 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:24.252998 28444 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:24.253119 27228 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:24.253607 28646 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:24.303386 27228 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:24.303949 27092 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 20:46:24.304020 28439 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:24.306600 28439 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 20:46:24.315888 28742 mvcc.cc:205] Tried to move back new op lower bound from 7023340684520783872 to 7023340683058622464. Current Snapshot: MvccSnapshot[applied={T|T < 7023340684492374016}]

INSERT report
    rows total: 2000
    time total: 164.33 ms
  time per row: 0.0821649 ms
I20240502 20:46:24.364662 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.133:45949
--local_ip_for_outbound_sockets=127.19.66.133
--tserver_master_addrs=127.19.66.190:43237
--webserver_port=42147
--webserver_interface=127.19.66.133
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 20:46:24.512687 28763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:24.512974 28763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:24.513025 28763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:24.516273 28763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:24.516352 28763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:24.516430 28763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.133
I20240502 20:46:24.520604 28763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.133:45949
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.19.66.133
--webserver_port=42147
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.28763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.133
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:24.522037 28763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:24.523689 28763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:24.535881 28768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:24.536336 28771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:24.536904 28769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:24.537905 28763 server_base.cc:1034] running on GCE node
I20240502 20:46:24.538614 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:24.539469 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:24.540679 28763 hybrid_clock.cc:648] HybridClock initialized: now 1714682784540666 us; error 36 us; skew 500 ppm
I20240502 20:46:24.543881 28763 webserver.cc:460] Webserver started at http://127.19.66.133:42147/ using document root <none> and password file <none>
I20240502 20:46:24.544713 28763 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:24.544800 28763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:24.550344 28763 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 20:46:24.553452 28777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:24.555295 28763 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
I20240502 20:46:24.555529 28763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1f40a11333e44ea0a34b6bc7618af522"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:24.556210 28763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:24.577035 28763 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:24.577896 28763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:24.578078 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:24.578703 28763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:24.579900 28763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:24.579962 28763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:24.580010 28763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:24.580027 28763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:24.593467 28763 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.133:45949
I20240502 20:46:24.593601 28895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.133:45949 every 8 connection(s)
I20240502 20:46:24.594986 28763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 20:46:24.605316 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 28763
I20240502 20:46:24.605988 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.134:46737
--local_ip_for_outbound_sockets=127.19.66.134
--tserver_master_addrs=127.19.66.190:43237
--webserver_port=46289
--webserver_interface=127.19.66.134
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:24.608954 28896 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:24.609500 28896 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:24.610808 28896 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:24.613005 26807 ts_manager.cc:194] Re-registered known tserver with Master: 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949)
I20240502 20:46:24.614023 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.133:53605
W20240502 20:46:24.764860 28899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:24.765255 28899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:24.765339 28899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:24.768839 28899 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:24.768945 28899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:24.769093 28899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.134
I20240502 20:46:24.773291 28899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.134:46737
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.19.66.134
--webserver_port=46289
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.28899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.134
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:24.774649 28899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:24.776278 28899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:24.788748 28904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:24.789121 28905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:24.789860 28907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:24.790699 28899 server_base.cc:1034] running on GCE node
I20240502 20:46:24.791424 28899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:24.792311 28899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:24.793509 28899 hybrid_clock.cc:648] HybridClock initialized: now 1714682784793483 us; error 47 us; skew 500 ppm
I20240502 20:46:24.797326 28899 webserver.cc:460] Webserver started at http://127.19.66.134:46289/ using document root <none> and password file <none>
I20240502 20:46:24.798197 28899 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:24.798300 28899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:24.803397 28899 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 20:46:24.806243 28913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:24.807862 28899 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 20:46:24.808036 28899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:24.808674 28899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:24.830619 28899 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:24.831610 28899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:24.831836 28899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:24.832700 28899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:24.834017 28899 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:24.834090 28899 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:24.834149 28899 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:24.834167 28899 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:24.847865 28899 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.134:46737
I20240502 20:46:24.847956 29028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.134:46737 every 8 connection(s)
I20240502 20:46:24.849606 28899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 20:46:24.859431 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 28899
I20240502 20:46:24.860118 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.135:46845
--local_ip_for_outbound_sockets=127.19.66.135
--tserver_master_addrs=127.19.66.190:43237
--webserver_port=38917
--webserver_interface=127.19.66.135
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:24.864077 29029 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:24.864483 29029 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:24.865535 29029 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:24.867425 26807 ts_manager.cc:194] Re-registered known tserver with Master: 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:24.868104 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.134:60747
W20240502 20:46:25.012928 29032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:25.013298 29032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:25.013408 29032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:25.018066 29032 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:25.018186 29032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:25.018323 29032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.135
I20240502 20:46:25.023422 29032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.135:46845
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.19.66.135
--webserver_port=38917
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.29032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.135
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:25.024734 29032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:25.026254 29032 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:25.038336 29038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:25.038856 29040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:25.039387 29037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:25.040119 29032 server_base.cc:1034] running on GCE node
I20240502 20:46:25.040642 29032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:25.041572 29032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:25.042865 29032 hybrid_clock.cc:648] HybridClock initialized: now 1714682785042837 us; error 46 us; skew 500 ppm
I20240502 20:46:25.046294 29032 webserver.cc:460] Webserver started at http://127.19.66.135:38917/ using document root <none> and password file <none>
I20240502 20:46:25.047359 29032 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:25.047540 29032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:25.053506 29032 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 20:46:25.056615 29047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:25.058748 29032 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 20:46:25.058970 29032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1bc8a142cfc7487bacc573afb0a92b74"
format_stamp: "Formatted at 2024-05-02 20:46:12 on dist-test-slave-gqqg"
I20240502 20:46:25.059660 29032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:25.078421 29032 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:25.079741 29032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:25.080000 29032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:25.080808 29032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:25.082062 29032 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:25.082121 29032 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:25.082171 29032 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:25.082191 29032 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:25.094552 29032 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.135:46845
I20240502 20:46:25.094575 29164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.135:46845 every 8 connection(s)
I20240502 20:46:25.096560 29032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 20:46:25.101120 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 29032
I20240502 20:46:25.101644 19722 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF51BWD/build/debug/bin/kudu
/tmp/dist-test-taskF51BWD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.136:41529
--local_ip_for_outbound_sockets=127.19.66.136
--tserver_master_addrs=127.19.66.190:43237
--webserver_port=39859
--webserver_interface=127.19.66.136
--builtin_ntp_servers=127.19.66.148:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 20:46:25.109452 29165 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:25.109820 29165 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:25.110742 29165 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:25.112339 26807 ts_manager.cc:194] Re-registered known tserver with Master: 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
I20240502 20:46:25.113332 26807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.135:60091
W20240502 20:46:25.272614 29168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:25.272935 29168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 20:46:25.272994 29168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:25.277874 29168 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 20:46:25.278123 29168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 20:46:25.278529 29168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.136
I20240502 20:46:25.283809 29168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.148:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.136:41529
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.19.66.136
--webserver_port=39859
--tserver_master_addrs=127.19.66.190:43237
--never_fsync=true
--heap_profile_path=/tmp/kudu.29168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.136
--log_dir=/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type DEBUG
built by None at 02 May 2024 20:15:58 UTC on efd1337a2aa3
build id 93
I20240502 20:46:25.285244 29168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:46:25.286655 29168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:46:25.297605 29174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:25.297909 29173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:46:25.298658 29176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:46:25.299284 29168 server_base.cc:1034] running on GCE node
I20240502 20:46:25.299928 29168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 20:46:25.300690 29168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 20:46:25.301878 29168 hybrid_clock.cc:648] HybridClock initialized: now 1714682785301857 us; error 43 us; skew 500 ppm
I20240502 20:46:25.304695 29168 webserver.cc:460] Webserver started at http://127.19.66.136:39859/ using document root <none> and password file <none>
I20240502 20:46:25.305649 29168 fs_manager.cc:359] Metadata directory not provided
I20240502 20:46:25.305788 29168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:46:25.311132 29168 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240502 20:46:25.314056 29182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:25.315860 29168 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:46:25.316041 29168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6706d1c0946d42a9bae62d719e1e5c31"
format_stamp: "Formatted at 2024-05-02 20:46:13 on dist-test-slave-gqqg"
I20240502 20:46:25.316612 29168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:46:25.348053 29168 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:46:25.348879 29168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:46:25.349107 29168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:46:25.349877 29168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:46:25.350997 29168 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:46:25.351058 29168 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:25.351127 29168 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:46:25.351156 29168 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:46:25.363673 29168 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.136:41529
I20240502 20:46:25.363696 29297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.136:41529 every 8 connection(s)
I20240502 20:46:25.365144 29168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 20:46:25.374282 19722 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF51BWD/build/debug/bin/kudu as pid 29168
I20240502 20:46:25.376587 29298 heartbeater.cc:347] Connected to a master server at 127.19.66.190:43237
I20240502 20:46:25.376890 29298 heartbeater.cc:464] Registering TS with master...
I20240502 20:46:25.378309 29298 heartbeater.cc:511] Master 127.19.66.190:43237 requested a full tablet report, sending...
I20240502 20:46:25.380164 26805 ts_manager.cc:194] Re-registered known tserver with Master: 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:25.381301 26805 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.136:54687
I20240502 20:46:25.618162 28896 heartbeater.cc:503] Master 127.19.66.190:43237 was elected leader, sending a full tablet report...
I20240502 20:46:25.872016 29029 heartbeater.cc:503] Master 127.19.66.190:43237 was elected leader, sending a full tablet report...
I20240502 20:46:25.890141 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:25.896224 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:25.908270 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:25.912288 29231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:25.912701 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:25.932188 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:25.933252 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:25.944715 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:25.950325 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:25.969924 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:25.971673 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:25.997241 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:25.997686 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.001104 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:26.026837 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:26.036762 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:26.064251 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.116302 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.117225 29165 heartbeater.cc:503] Master 127.19.66.190:43237 was elected leader, sending a full tablet report...
I20240502 20:46:26.120318 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.135056 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.142177 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.165603 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:26.169057 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.169057 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.178568 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:26.189497 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:26.203738 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:26.224434 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:26.239493 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.244287 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:26.249814 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.254084 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.278308 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.278452 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:26.280735 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:26.293515 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.320906 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:26.326704 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:26.333971 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:26.351246 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:26.385210 29298 heartbeater.cc:503] Master 127.19.66.190:43237 was elected leader, sending a full tablet report...
I20240502 20:46:26.656371 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.669835 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:26.674281 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.698169 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:26.699123 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:26.714809 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.728271 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.731271 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.733852 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.747238 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.754740 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.770248 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.779577 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:26.788298 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:26.795678 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:26.797951 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.807121 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:26.808972 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.824328 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:26.830140 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:26.837849 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:26.863761 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.867973 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:26.910152 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:26.938313 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:26.949620 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.951288 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:26.955670 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.959553 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.966750 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:26.968468 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:26.974560 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:26.976222 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:26.996340 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:26.999128 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.015538 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.022073 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.046911 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:27.052834 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:27.066340 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.268519 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.277637 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.283174 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.296072 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.302861 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:27.314620 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:27.319768 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:27.378930 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:27.434314 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.435381 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.440258 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.449652 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.463027 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:27.463389 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:27.471347 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:27.473408 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:27.524621 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.539084 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.543932 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.551420 27092 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:27.554937 27227 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.555083 27361 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.555790 29324 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.556249 29324 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.567795 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 20:46:27.582522 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1b9ff3c921d474fa5ad106d1c17ab00. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.582716 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:27.582743 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.595729 29324 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:27.599267 27361 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:27.599715 27228 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:27.604524 26811 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:27.614403 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:27.618558 27092 consensus_queue.cc:237] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:27.622033 26957 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.623075 27361 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.624289 30061 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.629606 30061 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:27.635418 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff9fef57f6eb490da536170a02b96212. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.640612 27092 consensus_queue.cc:237] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:27.644238 27228 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.644238 26960 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.644807 30061 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.646147 30061 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.648559 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.648972 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:27.649636 29324 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:27.651232 26960 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:27.652007 27228 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:27.654407 30061 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:27.656330 26806 catalog_manager.cc:5484] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:27.661383 27361 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:27.670331 26806 catalog_manager.cc:5484] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
W20240502 20:46:27.673475 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 135a08f1f2ac4707bbacee81ac73a19e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.683933 26957 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:27.684507 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.688208 27092 consensus_queue.cc:237] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:27.690343 27228 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.691066 30058 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:27.691349 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62ebc6576a4e4a108de55a4e9d7ff6b8. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.692263 26957 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.692754 30058 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.705741 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:27.706349 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.709450 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:27.713811 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:27.714809 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:27.738088 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.751543 30058 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:27.755079 26960 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:27.761250 26806 catalog_manager.cc:5484] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:27.765604 27227 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:27.785902 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:27.791450 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.796802 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:27.797319 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.797276 27092 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:27.799999 27361 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.800742 30061 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.803006 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.810154 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:27.811235 26960 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.817966 30064 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:27.818850 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f00c4b7954564820875a25da5834416a. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.832703 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:27.837556 27227 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:27.841161 27092 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.842139 30027 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.847019 27362 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.847543 30027 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.849212 29324 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:27.851915 27361 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:27.853091 26810 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:27.856603 26957 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:27.876739 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:27.886178 30027 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
W20240502 20:46:27.889418 27158 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf189c19164442e5af0827f1fb7dc085. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.893546 27092 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:27.898316 26810 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:27.898495 27362 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:27.908581 27227 consensus_queue.cc:237] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:27.918433 27362 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:27.920146 27092 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.921432 30028 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.922294 27227 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.922430 27092 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.923008 30029 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.923460 29570 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.923955 27362 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.924360 30028 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.926673 30027 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:27.927150 27362 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
W20240502 20:46:27.928314 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1545142b0bd40859bf33bf8686002ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.930586 26806 catalog_manager.cc:5484] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:27.930914 27092 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
W20240502 20:46:27.939182 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cf737e20ca74db1b94ccff5d60133ee. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.940579 29570 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:27.941256 27093 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:27.945030 26806 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:27.945768 27227 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:27.978749 27093 consensus_queue.cc:237] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:27.979280 27092 consensus_queue.cc:237] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:27.981662 27227 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.981621 26957 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.981850 27362 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:27.982129 27361 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:27.982481 27024 consensus_peers.cc:485] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367543df62e048268908eb0d84c14876. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.982784 30061 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.982993 30058 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:27.983071 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5704bdfe6c02437c8e9c8c3deaadb9e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.983304 30058 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.983378 30064 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:27.984746 27091 consensus_queue.cc:237] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
W20240502 20:46:27.987411 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 135a08f1f2ac4707bbacee81ac73a19e. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:27.987432 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 135a08f1f2ac4707bbacee81ac73a19e. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:27.988898 30187 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:27.989531 29324 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:27.990833 26957 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:27.991842 27362 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:27.992274 27361 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:27.992224 27227 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:27.994851 27228 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:27.995401 26806 catalog_manager.cc:5484] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:27.995740 26806 catalog_manager.cc:5484] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:27.995965 26957 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:27.996250 30058 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:27.997190 30058 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.005208 30061 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.009479 27228 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.013687 26806 catalog_manager.cc:5484] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 4 to 5, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.016520 27361 consensus_queue.cc:237] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.019353 26957 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.020828 26960 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.022859 29570 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.025799 29570 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.028550 26960 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.028929 27227 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
W20240502 20:46:28.029580 27295 consensus_peers.cc:485] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8deaa1f751034c15acd6ccaad37536c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.030552 26805 catalog_manager.cc:5484] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.028929 27228 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.031234 27361 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.031546 27091 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.032058 30029 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.033284 30027 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.035660 30027 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.037437 27228 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
W20240502 20:46:28.038678 27159 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a547561343304137a044ce11517f6007. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.039727 30027 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.044106 27091 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.046613 26806 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.048887 27361 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.063252 27361 consensus_queue.cc:237] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.065992 27091 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.066618 26960 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.067740 30029 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.068140 30147 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 20:46:28.068938 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a8ce1df86914b2caf2d083d97eda4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.072718 27227 consensus_queue.cc:237] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.076097 26959 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:28.076586 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 078a1591d9d74e31a0407aeab76911b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.076915 27361 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.077180 29790 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.077812 29790 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.081964 30027 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.084180 29570 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.084823 26960 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.087610 26960 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.087742 27091 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.088227 26806 catalog_manager.cc:5484] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.093273 26801 catalog_manager.cc:5484] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.127698 27361 consensus_queue.cc:237] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
W20240502 20:46:28.130147 27295 consensus_peers.cc:485] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14a7bead40fb4eb0bec6b00804b52595. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.130781 26960 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.131424 30029 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.131949 27091 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.132889 30029 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.134693 30029 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.137336 26960 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.137835 27091 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.139859 26812 catalog_manager.cc:5484] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.142385 30285 ts_tablet_manager.cc:902] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:28.143836 30285 tablet_copy_client.cc:308] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:28.148222 27228 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:28.148974 30295 ts_tablet_manager.cc:902] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:28.150050 30295 tablet_copy_client.cc:308] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
W20240502 20:46:28.151028 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a906569ab3b14520b4876bc8d5450f7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.151070 27091 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.151584 29790 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.152360 27361 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.154397 29790 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.157161 27361 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.159358 27113 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet d1b9ff3c921d474fa5ad106d1c17ab00 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:58553)
I20240502 20:46:28.159590 27113 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet d1b9ff3c921d474fa5ad106d1c17ab00 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:58553: session id = 6706d1c0946d42a9bae62d719e1e5c31-d1b9ff3c921d474fa5ad106d1c17ab00
I20240502 20:46:28.161625 27113 tablet_copy_source_session.cc:205] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.166651 27092 consensus_queue.cc:237] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:28.169507 26960 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.170783 27362 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:28.171293 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d44f271f166b4c288b76689a3e3764d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.171154 30248 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:28.172706 27362 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:28.175257 30194 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.177852 27091 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:28.179080 30285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b9ff3c921d474fa5ad106d1c17ab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.179910 30058 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.181906 26806 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.190776 30285 tablet_copy_client.cc:783] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.190999 27113 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet ff9fef57f6eb490da536170a02b96212 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:40783)
I20240502 20:46:28.191154 27113 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet ff9fef57f6eb490da536170a02b96212 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:40783: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-ff9fef57f6eb490da536170a02b96212
I20240502 20:46:28.191424 30285 tablet_copy_client.cc:647] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.192471 27113 tablet_copy_source_session.cc:205] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.194967 30285 tablet_copy_client.cc:523] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.198218 30285 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:28.200749 30295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9fef57f6eb490da536170a02b96212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.205559 27091 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.206079 30061 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:28.208137 27362 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:28.208770 27228 consensus_queue.cc:237] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:28.209103 26960 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.210536 30187 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.211544 26960 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:28.212422 26959 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 20:46:28.213008 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 078a1591d9d74e31a0407aeab76911b4. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:28.213413 27025 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdd2311ec0b645f5a79098a32c6deba2. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.213376 30248 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:28.213955 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 078a1591d9d74e31a0407aeab76911b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.215332 27362 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:28.215901 30248 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.216708 30248 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:28.219709 26960 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:28.220718 27228 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.221509 30285 log.cc:824] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:28.222098 26801 catalog_manager.cc:5484] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:28.223896 26805 catalog_manager.cc:5484] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.223985 30194 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.227581 27361 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:28.239441 30349 ts_tablet_manager.cc:902] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:28.240295 30349 tablet_copy_client.cc:308] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:28.241014 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:58553)
I20240502 20:46:28.241190 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:58553: session id = 6706d1c0946d42a9bae62d719e1e5c31-62ebc6576a4e4a108de55a4e9d7ff6b8
I20240502 20:46:28.241506 30058 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.242445 27112 tablet_copy_source_session.cc:205] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.242451 27228 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.246666 26806 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.249409 26959 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.261863 30349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.265493 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.267153 27228 consensus_queue.cc:237] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.269793 26959 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.270004 27091 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.270579 29790 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.271061 30028 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.275450 30285 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.276176 30295 tablet_copy_client.cc:783] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.276229 30285 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:28.276649 30295 tablet_copy_client.cc:647] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.276710 30285 ts_tablet_manager.cc:1367] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.079s	user 0.050s	sys 0.000s
I20240502 20:46:28.280376 30285 raft_consensus.cc:348] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.280761 30285 raft_consensus.cc:729] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:28.280562 30028 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
W20240502 20:46:28.280994 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b466a750cc6b4382947282d8aa6aeb1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.281273 30285 consensus_queue.cc:260] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.282691 30285 ts_tablet_manager.cc:1398] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240502 20:46:28.283360 30349 tablet_copy_client.cc:783] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.283558 30349 tablet_copy_client.cc:647] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.284888 26959 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.287068 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-d1b9ff3c921d474fa5ad106d1c17ab00 received from {username='slave'} at 127.19.66.136:58553
I20240502 20:46:28.287189 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-d1b9ff3c921d474fa5ad106d1c17ab00 on tablet d1b9ff3c921d474fa5ad106d1c17ab00 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:28.287453 27091 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.288215 30295 tablet_copy_client.cc:523] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.289819 26806 catalog_manager.cc:5484] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.292197 30295 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:28.319808 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:28.325179 30385 ts_tablet_manager.cc:902] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:28.326786 30385 tablet_copy_client.cc:308] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:28.326977 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:28.328749 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.330526 30384 ts_tablet_manager.cc:902] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:28.331584 30384 tablet_copy_client.cc:308] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:28.341362 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.349148 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:28.363535 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet f00c4b7954564820875a25da5834416a from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:48853)
I20240502 20:46:28.363757 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet f00c4b7954564820875a25da5834416a from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:48853: session id = 1f40a11333e44ea0a34b6bc7618af522-f00c4b7954564820875a25da5834416a
I20240502 20:46:28.365141 27112 tablet_copy_source_session.cc:205] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.365550 30295 log.cc:824] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:28.369561 30384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00c4b7954564820875a25da5834416a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.376982 27091 consensus_queue.cc:237] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.380111 27361 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 20:46:28.380299 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d44f271f166b4c288b76689a3e3764d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.380615 26959 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:28.380928 30194 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.382258 30349 tablet_copy_client.cc:523] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.384934 30349 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:28.387189 26959 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.388557 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 20:46:28.390990 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d44f271f166b4c288b76689a3e3764d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.390974 27228 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.391255 30194 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.391278 30064 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.391796 28374 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.392473 27360 consensus_queue.cc:237] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.395128 27091 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.395965 27362 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.397084 26801 catalog_manager.cc:5484] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.402530 26959 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.404253 26960 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:28.405226 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f00d12e425284c36ae8baed56c7cfaf6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.402551 28374 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20240502 20:46:28.408092 30284 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.409704 27228 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.410291 30284 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.420922 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.422623 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.424376 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.425411 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:28.433271 30427 ts_tablet_manager.cc:902] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:28.434108 30427 tablet_copy_client.cc:308] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:28.440043 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet f1545142b0bd40859bf33bf8686002ae from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:39121)
I20240502 20:46:28.440320 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet f1545142b0bd40859bf33bf8686002ae from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:39121: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-f1545142b0bd40859bf33bf8686002ae
I20240502 20:46:28.442276 27249 tablet_copy_source_session.cc:205] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.444860 30427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1545142b0bd40859bf33bf8686002ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.446122 30349 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.446704 30349 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:28.447018 30349 ts_tablet_manager.cc:1367] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.062s	user 0.032s	sys 0.000s
I20240502 20:46:28.447907 30349 raft_consensus.cc:348] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.448163 30349 raft_consensus.cc:729] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:28.448395 30349 consensus_queue.cc:260] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.448904 30427 tablet_copy_client.cc:783] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.449132 30427 tablet_copy_client.cc:647] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.449942 30349 ts_tablet_manager.cc:1398] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:46:28.450661 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-62ebc6576a4e4a108de55a4e9d7ff6b8 received from {username='slave'} at 127.19.66.136:58553
I20240502 20:46:28.450757 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-62ebc6576a4e4a108de55a4e9d7ff6b8 on tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:28.451084 30285 ts_tablet_manager.cc:902] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:28.451704 30285 tablet_copy_client.cc:308] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:28.453097 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet 135a08f1f2ac4707bbacee81ac73a19e from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:58553)
I20240502 20:46:28.453259 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet 135a08f1f2ac4707bbacee81ac73a19e from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:58553: session id = 6706d1c0946d42a9bae62d719e1e5c31-135a08f1f2ac4707bbacee81ac73a19e
I20240502 20:46:28.454423 27112 tablet_copy_source_session.cc:205] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.456526 30285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135a08f1f2ac4707bbacee81ac73a19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.459794 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 5cf737e20ca74db1b94ccff5d60133ee from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:58959)
I20240502 20:46:28.459991 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 5cf737e20ca74db1b94ccff5d60133ee from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:58959: session id = 1bc8a142cfc7487bacc573afb0a92b74-5cf737e20ca74db1b94ccff5d60133ee
I20240502 20:46:28.460433 28374 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.461903 27382 tablet_copy_source_session.cc:205] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.464196 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:28.465273 27228 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.465859 26806 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.470353 30285 tablet_copy_client.cc:783] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.470716 30285 tablet_copy_client.cc:647] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.479032 30285 tablet_copy_client.cc:523] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.479919 30385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf737e20ca74db1b94ccff5d60133ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.480755 30285 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:28.485734 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:28.486526 30284 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.489403 27228 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.493324 30444 ts_tablet_manager.cc:902] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:28.493184 26805 catalog_manager.cc:5484] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.494181 30444 tablet_copy_client.cc:308] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:28.500808 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:28.503444 27092 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.508999 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:52059)
I20240502 20:46:28.509199 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:52059: session id = 1bc8a142cfc7487bacc573afb0a92b74-5704bdfe6c02437c8e9c8c3deaadb9e3
I20240502 20:46:28.510321 27112 tablet_copy_source_session.cc:205] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.512732 30444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.529054 30385 tablet_copy_client.cc:783] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.529500 30385 tablet_copy_client.cc:647] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.531944 30385 tablet_copy_client.cc:523] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.534250 30444 tablet_copy_client.cc:783] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.534576 30444 tablet_copy_client.cc:647] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
W20240502 20:46:28.534543 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af693f5816e046f592fbd9b2e7ef0ad9. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.537025 30385 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:28.536952 26959 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.541515 30444 tablet_copy_client.cc:523] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.543705 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.543879 30444 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:28.545480 30437 ts_tablet_manager.cc:902] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:28.546551 30437 tablet_copy_client.cc:308] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:28.546756 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:28.548223 30384 tablet_copy_client.cc:783] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.548625 30384 tablet_copy_client.cc:647] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.546833 30443 ts_tablet_manager.cc:902] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:28.550197 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 6a8ce1df86914b2caf2d083d97eda4e6 from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:58959)
I20240502 20:46:28.550330 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 6a8ce1df86914b2caf2d083d97eda4e6 from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:58959: session id = 1bc8a142cfc7487bacc573afb0a92b74-6a8ce1df86914b2caf2d083d97eda4e6
I20240502 20:46:28.550873 30443 tablet_copy_client.cc:308] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:28.551549 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet 135a08f1f2ac4707bbacee81ac73a19e from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:52059)
I20240502 20:46:28.551680 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet 135a08f1f2ac4707bbacee81ac73a19e from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:52059: session id = 1bc8a142cfc7487bacc573afb0a92b74-135a08f1f2ac4707bbacee81ac73a19e
I20240502 20:46:28.552843 27112 tablet_copy_source_session.cc:205] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.554150 30443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135a08f1f2ac4707bbacee81ac73a19e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.554416 27382 tablet_copy_source_session.cc:205] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.555289 30437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8ce1df86914b2caf2d083d97eda4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.558756 30411 ts_tablet_manager.cc:902] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:28.558761 30436 ts_tablet_manager.cc:902] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:28.559588 30384 tablet_copy_client.cc:523] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.562211 30437 tablet_copy_client.cc:783] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.562649 30437 tablet_copy_client.cc:647] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.562670 30384 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:28.562856 30443 tablet_copy_client.cc:783] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.563022 30443 tablet_copy_client.cc:647] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.563870 30436 tablet_copy_client.cc:308] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:28.564960 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet 367543df62e048268908eb0d84c14876 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:48853)
I20240502 20:46:28.565179 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet 367543df62e048268908eb0d84c14876 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:48853: session id = 1f40a11333e44ea0a34b6bc7618af522-367543df62e048268908eb0d84c14876
I20240502 20:46:28.565614 30427 tablet_copy_client.cc:523] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.567148 27112 tablet_copy_source_session.cc:205] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.567628 30427 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:28.568549 30436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367543df62e048268908eb0d84c14876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.571681 30436 tablet_copy_client.cc:783] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.571981 30436 tablet_copy_client.cc:647] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.573249 30411 tablet_copy_client.cc:308] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:28.573715 27362 consensus_queue.cc:237] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.575804 26959 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:28.576289 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 843a26fc48844bfd82911b6a27833023. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.576351 27228 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.576809 29570 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.577035 30147 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.579068 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet bf189c19164442e5af0827f1fb7dc085 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:35399)
I20240502 20:46:28.579205 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet bf189c19164442e5af0827f1fb7dc085 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:35399: session id = 1f40a11333e44ea0a34b6bc7618af522-bf189c19164442e5af0827f1fb7dc085
I20240502 20:46:28.580416 27249 tablet_copy_source_session.cc:205] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.581763 30411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf189c19164442e5af0827f1fb7dc085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.584440 30437 tablet_copy_client.cc:523] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.585379 30411 tablet_copy_client.cc:783] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.585593 30411 tablet_copy_client.cc:647] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.586542 30437 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:28.593853 30437 log.cc:824] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:28.594904 30465 ts_tablet_manager.cc:902] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:28.595825 30465 tablet_copy_client.cc:308] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:28.601861 29252 raft_consensus.cc:1203] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:28.603307 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 8deaa1f751034c15acd6ccaad37536c6 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:53843)
I20240502 20:46:28.603454 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 8deaa1f751034c15acd6ccaad37536c6 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:53843: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-8deaa1f751034c15acd6ccaad37536c6
I20240502 20:46:28.605252 27382 tablet_copy_source_session.cc:205] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.605540 30443 tablet_copy_client.cc:523] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.607474 30443 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:28.613675 27091 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
W20240502 20:46:28.615764 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdd2311ec0b645f5a79098a32c6deba2. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.616160 26960 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:28.616669 30058 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.617240 30029 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.617440 27227 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:28.618018 26959 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
W20240502 20:46:28.618367 27025 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdd2311ec0b645f5a79098a32c6deba2. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.619197 30058 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.621960 30058 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.622741 26806 catalog_manager.cc:5484] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.623790 27227 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.624429 30465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8deaa1f751034c15acd6ccaad37536c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.627883 30465 tablet_copy_client.cc:783] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.628649 26806 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.628805 29252 raft_consensus.cc:1203] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:28.628885 26960 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.629822 30465 tablet_copy_client.cc:647] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.634187 27227 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.637439 30349 ts_tablet_manager.cc:902] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:28.637979 30349 tablet_copy_client.cc:308] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:28.648204 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet a547561343304137a044ce11517f6007 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:35339)
I20240502 20:46:28.648366 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet a547561343304137a044ce11517f6007 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:35339: session id = 6706d1c0946d42a9bae62d719e1e5c31-a547561343304137a044ce11517f6007
I20240502 20:46:28.649479 27249 tablet_copy_source_session.cc:205] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.650772 30349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a547561343304137a044ce11517f6007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.654564 30349 tablet_copy_client.cc:783] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.654851 30349 tablet_copy_client.cc:647] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.637738 30489 ts_tablet_manager.cc:902] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:28.665385 30489 tablet_copy_client.cc:308] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:28.666985 27381 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 14a7bead40fb4eb0bec6b00804b52595 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:53843)
I20240502 20:46:28.667119 27381 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 14a7bead40fb4eb0bec6b00804b52595 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:53843: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-14a7bead40fb4eb0bec6b00804b52595
I20240502 20:46:28.660811 30427 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.669157 27381 tablet_copy_source_session.cc:205] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.670609 30427 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:28.670996 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a7bead40fb4eb0bec6b00804b52595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.671072 30427 ts_tablet_manager.cc:1367] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.104s	user 0.030s	sys 0.010s
I20240502 20:46:28.671355 30465 tablet_copy_client.cc:523] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.674865 30489 tablet_copy_client.cc:783] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.675199 30489 tablet_copy_client.cc:647] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.680212 30489 tablet_copy_client.cc:523] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.681003 30465 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:28.683002 30489 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:28.687163 30349 tablet_copy_client.cc:523] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.689193 30427 raft_consensus.cc:348] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.689711 30349 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:28.689617 30427 raft_consensus.cc:729] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:28.693145 30427 consensus_queue.cc:260] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.694478 30427 ts_tablet_manager.cc:1398] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20240502 20:46:28.696254 30501 ts_tablet_manager.cc:902] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:28.697185 30501 tablet_copy_client.cc:308] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:28.698012 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet 078a1591d9d74e31a0407aeab76911b4 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:35399)
I20240502 20:46:28.698156 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet 078a1591d9d74e31a0407aeab76911b4 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:35399: session id = 1f40a11333e44ea0a34b6bc7618af522-078a1591d9d74e31a0407aeab76911b4
I20240502 20:46:28.699975 27249 tablet_copy_source_session.cc:205] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.709614 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-f1545142b0bd40859bf33bf8686002ae received from {username='slave'} at 127.19.66.134:39121
I20240502 20:46:28.709764 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-f1545142b0bd40859bf33bf8686002ae on tablet f1545142b0bd40859bf33bf8686002ae with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:28.711766 30384 log.cc:824] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: Log is configured to *not* fsync() on all Append() calls
I20240502 20:46:28.712725 30411 tablet_copy_client.cc:523] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.716308 30411 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:28.722635 30443 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.723395 30443 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:28.723924 30436 tablet_copy_client.cc:523] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.724004 30443 ts_tablet_manager.cc:1367] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.117s	user 0.033s	sys 0.006s
I20240502 20:46:28.726037 30436 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:28.731554 30501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078a1591d9d74e31a0407aeab76911b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.739403 30501 tablet_copy_client.cc:783] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.739717 30501 tablet_copy_client.cc:647] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.742086 30513 ts_tablet_manager.cc:902] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:28.742983 30513 tablet_copy_client.cc:308] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:28.743573 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet a906569ab3b14520b4876bc8d5450f7d from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:35399)
I20240502 20:46:28.743695 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet a906569ab3b14520b4876bc8d5450f7d from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:35399: session id = 1f40a11333e44ea0a34b6bc7618af522-a906569ab3b14520b4876bc8d5450f7d
I20240502 20:46:28.744793 27249 tablet_copy_source_session.cc:205] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.747692 30513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a906569ab3b14520b4876bc8d5450f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.755283 27227 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.754143 30437 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.755937 30437 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:28.756456 30437 ts_tablet_manager.cc:1367] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.170s	user 0.042s	sys 0.005s
W20240502 20:46:28.758937 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf189c19164442e5af0827f1fb7dc085. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.760061 27362 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:28.761178 30248 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.764557 30248 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.768752 27362 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.771492 30427 ts_tablet_manager.cc:902] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:28.772056 30427 tablet_copy_client.cc:308] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:28.772756 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet 078a1591d9d74e31a0407aeab76911b4 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:39121)
I20240502 20:46:28.772881 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet 078a1591d9d74e31a0407aeab76911b4 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:39121: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-078a1591d9d74e31a0407aeab76911b4
I20240502 20:46:28.773962 27249 tablet_copy_source_session.cc:205] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.774233 26806 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 4 to 5, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.775012 30437 raft_consensus.cc:348] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.775391 30427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078a1591d9d74e31a0407aeab76911b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.775400 30437 raft_consensus.cc:729] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:28.775892 30437 consensus_queue.cc:260] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.777254 30437 ts_tablet_manager.cc:1398] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20240502 20:46:28.779400 27381 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-6a8ce1df86914b2caf2d083d97eda4e6 received from {username='slave'} at 127.19.66.135:58959
I20240502 20:46:28.779515 27381 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-6a8ce1df86914b2caf2d083d97eda4e6 on tablet 6a8ce1df86914b2caf2d083d97eda4e6 with peer 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:28.779934 27158 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.780354 30443 raft_consensus.cc:348] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.780565 30443 raft_consensus.cc:729] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:28.780737 30443 consensus_queue.cc:260] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.781361 30443 ts_tablet_manager.cc:1398] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.057s	user 0.004s	sys 0.000s
I20240502 20:46:28.784467 27091 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:28.785110 30248 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.786897 27091 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.790313 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-135a08f1f2ac4707bbacee81ac73a19e received from {username='slave'} at 127.19.66.135:52059
I20240502 20:46:28.790441 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-135a08f1f2ac4707bbacee81ac73a19e on tablet 135a08f1f2ac4707bbacee81ac73a19e with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:28.793705 30427 tablet_copy_client.cc:783] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.794103 30427 tablet_copy_client.cc:647] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.796828 30427 tablet_copy_client.cc:523] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.798990 30427 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:28.804216 30285 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.804621 30285 tablet_bootstrap.cc:492] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:28.805109 30285 ts_tablet_manager.cc:1367] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.324s	user 0.036s	sys 0.000s
I20240502 20:46:28.805748 30285 raft_consensus.cc:348] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.805900 30285 raft_consensus.cc:729] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:28.806115 30285 consensus_queue.cc:260] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.806571 30285 ts_tablet_manager.cc:1398] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:46:28.807482 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-135a08f1f2ac4707bbacee81ac73a19e received from {username='slave'} at 127.19.66.136:58553
I20240502 20:46:28.807582 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-135a08f1f2ac4707bbacee81ac73a19e on tablet 135a08f1f2ac4707bbacee81ac73a19e with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:28.815601 27227 consensus_queue.cc:237] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:28.816823 30501 tablet_copy_client.cc:523] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.818157 27362 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:28.818724 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5f81e46ab1f48fc8a88495b42f83640. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.819437 26959 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:28.819955 30228 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.820407 30228 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:28.824527 30501 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:28.833055 30295 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.833487 30295 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:28.833737 30295 ts_tablet_manager.cc:1367] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.542s	user 0.043s	sys 0.005s
I20240502 20:46:28.835897 30465 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.836308 30465 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:28.836535 30465 ts_tablet_manager.cc:1367] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.156s	user 0.027s	sys 0.007s
I20240502 20:46:28.836875 30489 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.837196 30489 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:28.837399 30489 ts_tablet_manager.cc:1367] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.155s	user 0.031s	sys 0.008s
I20240502 20:46:28.837365 30465 raft_consensus.cc:348] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.837774 30465 raft_consensus.cc:729] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:28.838021 30465 consensus_queue.cc:260] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.838850 30465 ts_tablet_manager.cc:1398] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 20:46:28.840111 30513 tablet_copy_client.cc:783] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.840405 30513 tablet_copy_client.cc:647] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.841866 30295 raft_consensus.cc:348] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.842049 30295 raft_consensus.cc:729] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:28.838300 30489 raft_consensus.cc:348] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.842239 30489 raft_consensus.cc:729] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:28.842309 30295 consensus_queue.cc:260] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.842407 30489 consensus_queue.cc:260] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.842865 30295 ts_tablet_manager.cc:1398] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240502 20:46:28.844729 30489 ts_tablet_manager.cc:1398] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240502 20:46:28.846319 27381 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-8deaa1f751034c15acd6ccaad37536c6 received from {username='slave'} at 127.19.66.134:53843
I20240502 20:46:28.846457 27381 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-8deaa1f751034c15acd6ccaad37536c6 on tablet 8deaa1f751034c15acd6ccaad37536c6 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:28.846733 30513 tablet_copy_client.cc:523] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.846930 30285 ts_tablet_manager.cc:902] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:28.847551 30285 tablet_copy_client.cc:308] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:38835
I20240502 20:46:28.849916 30513 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:28.857378 29252 raft_consensus.cc:1203] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 20:46:28.861510 27381 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-14a7bead40fb4eb0bec6b00804b52595 received from {username='slave'} at 127.19.66.134:53843
I20240502 20:46:28.861655 27381 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-14a7bead40fb4eb0bec6b00804b52595 on tablet 14a7bead40fb4eb0bec6b00804b52595 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:28.861701 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-ff9fef57f6eb490da536170a02b96212 received from {username='slave'} at 127.19.66.134:40783
I20240502 20:46:28.861790 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-ff9fef57f6eb490da536170a02b96212 on tablet ff9fef57f6eb490da536170a02b96212 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:28.863775 26980 tablet_copy_service.cc:140] P 297de823852d4a8da44791ab03131cd3: Received BeginTabletCopySession request for tablet af693f5816e046f592fbd9b2e7ef0ad9 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:35489)
I20240502 20:46:28.864032 26980 tablet_copy_service.cc:161] P 297de823852d4a8da44791ab03131cd3: Beginning new tablet copy session on tablet af693f5816e046f592fbd9b2e7ef0ad9 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:35489: session id = 6706d1c0946d42a9bae62d719e1e5c31-af693f5816e046f592fbd9b2e7ef0ad9
I20240502 20:46:28.866075 26980 tablet_copy_source_session.cc:205] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.866148 30437 ts_tablet_manager.cc:902] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:28.866912 30437 tablet_copy_client.cc:308] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:28.869184 30443 ts_tablet_manager.cc:902] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:28.869794 30443 tablet_copy_client.cc:308] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:28.870342 27381 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet f00d12e425284c36ae8baed56c7cfaf6 from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:58959)
I20240502 20:46:28.870483 27381 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet f00d12e425284c36ae8baed56c7cfaf6 from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:58959: session id = 1bc8a142cfc7487bacc573afb0a92b74-f00d12e425284c36ae8baed56c7cfaf6
I20240502 20:46:28.871647 27381 tablet_copy_source_session.cc:205] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.872728 30443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00d12e425284c36ae8baed56c7cfaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.877179 30436 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.877303 30443 tablet_copy_client.cc:783] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.877540 30443 tablet_copy_client.cc:647] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.879714 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet b466a750cc6b4382947282d8aa6aeb1f from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:49537)
I20240502 20:46:28.879914 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet b466a750cc6b4382947282d8aa6aeb1f from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:49537: session id = 1bc8a142cfc7487bacc573afb0a92b74-b466a750cc6b4382947282d8aa6aeb1f
I20240502 20:46:28.881017 27249 tablet_copy_source_session.cc:205] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.884281 27092 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.885325 30436 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:28.885952 30436 ts_tablet_manager.cc:1367] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.160s	user 0.035s	sys 0.004s
I20240502 20:46:28.886826 27360 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:28.887383 30187 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.887698 30437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b466a750cc6b4382947282d8aa6aeb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.889575 30411 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.891332 29117 raft_consensus.cc:1203] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 20:46:28.891912 30444 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.892289 30444 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:28.892511 30444 ts_tablet_manager.cc:1367] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.349s	user 0.032s	sys 0.000s
I20240502 20:46:28.893716 30411 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:28.894044 30411 ts_tablet_manager.cc:1367] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.179s	user 0.040s	sys 0.004s
I20240502 20:46:28.894847 30436 raft_consensus.cc:348] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
W20240502 20:46:28.895838 27026 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f00c4b7954564820875a25da5834416a. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.896054 30436 raft_consensus.cc:729] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
W20240502 20:46:28.896201 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.897320 30411 raft_consensus.cc:348] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:28.897554 26960 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:28.897992 30411 raft_consensus.cc:729] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:28.898047 30194 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.898279 30411 consensus_queue.cc:260] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:28.899680 30411 ts_tablet_manager.cc:1398] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 20:46:28.897814 30436 consensus_queue.cc:260] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:28.900830 30436 ts_tablet_manager.cc:1398] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20240502 20:46:28.901499 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-367543df62e048268908eb0d84c14876 received from {username='slave'} at 127.19.66.133:48853
I20240502 20:46:28.901638 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-367543df62e048268908eb0d84c14876 on tablet 367543df62e048268908eb0d84c14876 with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:28.901955 30437 tablet_copy_client.cc:783] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.902273 30437 tablet_copy_client.cc:647] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.903112 30385 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.903506 30385 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:28.903752 30385 ts_tablet_manager.cc:1367] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.367s	user 0.035s	sys 0.000s
I20240502 20:46:28.904294 30443 tablet_copy_client.cc:523] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.905637 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-bf189c19164442e5af0827f1fb7dc085 received from {username='slave'} at 127.19.66.133:35399
I20240502 20:46:28.905761 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-bf189c19164442e5af0827f1fb7dc085 on tablet bf189c19164442e5af0827f1fb7dc085 with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:28.909276 30443 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:28.917146 30248 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.918037 27227 consensus_queue.cc:237] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.919163 30444 raft_consensus.cc:348] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.919401 30444 raft_consensus.cc:729] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:28.919564 30444 consensus_queue.cc:260] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.919951 30444 ts_tablet_manager.cc:1398] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.027s	user 0.003s	sys 0.000s
I20240502 20:46:28.920872 30385 raft_consensus.cc:348] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.921013 30385 raft_consensus.cc:729] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:28.921303 30385 consensus_queue.cc:260] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:28.921681 30385 ts_tablet_manager.cc:1398] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.018s	user 0.002s	sys 0.000s
I20240502 20:46:28.922212 28980 raft_consensus.cc:1203] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:28.923439 26805 catalog_manager.cc:5484] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.923831 26959 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.927109 27361 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:28.927819 26957 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:28.928202 30027 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:46:28.928710 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.929527 27381 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-5cf737e20ca74db1b94ccff5d60133ee received from {username='slave'} at 127.19.66.135:58959
I20240502 20:46:28.929634 27381 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-5cf737e20ca74db1b94ccff5d60133ee on tablet 5cf737e20ca74db1b94ccff5d60133ee with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:28.929710 30228 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:46:28.931648 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.933393 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-5704bdfe6c02437c8e9c8c3deaadb9e3 received from {username='slave'} at 127.19.66.135:52059
I20240502 20:46:28.933518 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-5704bdfe6c02437c8e9c8c3deaadb9e3 on tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 with peer 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:28.935150 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 078a1591d9d74e31a0407aeab76911b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.935287 30248 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.938228 27361 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.940213 30513 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:28.940491 26957 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.940714 30513 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:28.940943 30513 ts_tablet_manager.cc:1367] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.091s	user 0.030s	sys 0.003s
I20240502 20:46:28.941651 30513 raft_consensus.cc:348] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:28.941912 30513 raft_consensus.cc:729] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:28.942111 30513 consensus_queue.cc:260] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:28.942643 30513 ts_tablet_manager.cc:1398] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 20:46:28.942616 26812 catalog_manager.cc:5484] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 5 to 6, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.950745 30437 tablet_copy_client.cc:523] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.952041 27362 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:28.952751 30437 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:28.953550 28848 raft_consensus.cc:1203] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:28.956851 30285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693f5816e046f592fbd9b2e7ef0ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.960346 30295 ts_tablet_manager.cc:902] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:28.961020 30285 tablet_copy_client.cc:783] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.961212 30295 tablet_copy_client.cc:308] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:28.961380 30285 tablet_copy_client.cc:647] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.962308 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet d44f271f166b4c288b76689a3e3764d3 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:40783)
I20240502 20:46:28.962510 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet d44f271f166b4c288b76689a3e3764d3 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:40783: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-d44f271f166b4c288b76689a3e3764d3
I20240502 20:46:28.963930 27112 tablet_copy_source_session.cc:205] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:28.965574 30295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44f271f166b4c288b76689a3e3764d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:28.967108 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-a906569ab3b14520b4876bc8d5450f7d received from {username='slave'} at 127.19.66.133:35399
I20240502 20:46:28.967233 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-a906569ab3b14520b4876bc8d5450f7d on tablet a906569ab3b14520b4876bc8d5450f7d with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:28.970165 30295 tablet_copy_client.cc:783] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:28.970393 30295 tablet_copy_client.cc:647] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:28.972800 27092 consensus_queue.cc:237] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:28.976256 27227 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:28.977898 30058 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.979601 28848 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:28.979674 30295 tablet_copy_client.cc:523] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:28.981241 29116 raft_consensus.cc:1203] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:28.982396 29116 raft_consensus.cc:1203] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240502 20:46:28.983240 27024 consensus_peers.cc:485] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367543df62e048268908eb0d84c14876. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:28.983433 30187 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:28.987524 30064 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.987654 29324 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.988243 27360 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:28.990424 27227 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:28.987355 30295 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:28.998791 26806 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 4 to 5, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:28.999269 26806 catalog_manager.cc:5484] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:29.003574 30513 ts_tablet_manager.cc:902] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:29.004036 26960 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.004386 30513 tablet_copy_client.cc:308] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:29.005205 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet d44f271f166b4c288b76689a3e3764d3 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:48853)
I20240502 20:46:29.005352 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet d44f271f166b4c288b76689a3e3764d3 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:48853: session id = 1f40a11333e44ea0a34b6bc7618af522-d44f271f166b4c288b76689a3e3764d3
I20240502 20:46:29.006132 27362 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.006561 27112 tablet_copy_source_session.cc:205] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.006657 30187 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.007905 30513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44f271f166b4c288b76689a3e3764d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.007992 27360 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:29.009152 30285 tablet_copy_client.cc:523] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.011284 30513 tablet_copy_client.cc:783] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.011600 30285 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:29.011786 30513 tablet_copy_client.cc:647] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.030103 30489 ts_tablet_manager.cc:902] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:29.030838 30489 tablet_copy_client.cc:308] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:29.031409 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet fdd2311ec0b645f5a79098a32c6deba2 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:40783)
I20240502 20:46:29.031553 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet fdd2311ec0b645f5a79098a32c6deba2 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:40783: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-fdd2311ec0b645f5a79098a32c6deba2
I20240502 20:46:29.031824 30513 tablet_copy_client.cc:523] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.032804 27112 tablet_copy_source_session.cc:205] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.033995 30513 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:29.034706 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd2311ec0b645f5a79098a32c6deba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.038106 27092 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.038581 30489 tablet_copy_client.cc:783] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.038815 30489 tablet_copy_client.cc:647] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
W20240502 20:46:29.041399 27025 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdd2311ec0b645f5a79098a32c6deba2. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.042769 26960 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 20:46:29.043521 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.042174 27227 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 20:46:29.043896 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdd2311ec0b645f5a79098a32c6deba2. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.045650 30437 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.046005 30437 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:29.046085 30187 raft_consensus.cc:1050] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 to VOTER
I20240502 20:46:29.046213 30437 ts_tablet_manager.cc:1367] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.094s	user 0.028s	sys 0.004s
I20240502 20:46:29.046756 30437 raft_consensus.cc:348] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.046878 30437 raft_consensus.cc:729] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:29.046975 30437 consensus_queue.cc:260] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.046914 30187 consensus_queue.cc:237] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:29.050495 27360 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.053866 30187 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.044551 30058 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.060233 30058 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.061013 30437 ts_tablet_manager.cc:1398] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.000s
I20240502 20:46:29.062809 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-b466a750cc6b4382947282d8aa6aeb1f received from {username='slave'} at 127.19.66.135:49537
I20240502 20:46:29.062978 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-b466a750cc6b4382947282d8aa6aeb1f on tablet b466a750cc6b4382947282d8aa6aeb1f with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:29.066037 30193 raft_consensus.cc:1050] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:29.067198 30193 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:29.070078 29252 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.070151 27092 consensus_queue.cc:237] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.070561 30443 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.070765 29324 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.070991 30443 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:29.071219 30443 ts_tablet_manager.cc:1367] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.164s	user 0.037s	sys 0.000s
I20240502 20:46:29.071828 30443 raft_consensus.cc:348] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.071969 30443 raft_consensus.cc:729] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:29.072077 30443 consensus_queue.cc:260] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.072635 27226 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.073626 27228 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.074360 27361 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.077111 27228 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:29.077525 30194 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.077771 29570 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:29.077780 26806 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.079047 28848 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:29.080443 29251 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.082823 27092 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 20:46:29.083351 27024 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62ebc6576a4e4a108de55a4e9d7ff6b8. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:29.083560 27024 consensus_peers.cc:485] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367543df62e048268908eb0d84c14876. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.084306 28847 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.084290 26957 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.087390 29324 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.088049 26960 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.090167 30194 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.091977 29570 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:29.092631 30489 tablet_copy_client.cc:523] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.092854 30194 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.094863 30489 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:29.096437 29324 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.098037 27228 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.100311 30443 ts_tablet_manager.cc:1398] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.029s	user 0.001s	sys 0.000s
I20240502 20:46:29.100634 30193 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.101411 27381 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-f00d12e425284c36ae8baed56c7cfaf6 received from {username='slave'} at 127.19.66.135:58959
I20240502 20:46:29.101517 27381 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-f00d12e425284c36ae8baed56c7cfaf6 on tablet f00d12e425284c36ae8baed56c7cfaf6 with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:29.107110 27227 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.107854 30193 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.109174 27362 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.109779 30193 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.111171 28972 raft_consensus.cc:1203] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:29.112779 30187 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:29.116627 30427 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.122751 30147 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.123348 27226 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.130406 26812 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:29.135011 30193 raft_consensus.cc:1013] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: attempt to promote peer 6706d1c0946d42a9bae62d719e1e5c31: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 20:46:29.136034 27227 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
W20240502 20:46:29.139899 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5586b9e60b544aed87c24f7b3ce0198c. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.144814 27091 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.152035 30194 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.155264 30349 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.155800 30349 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:29.156188 30349 ts_tablet_manager.cc:1367] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.467s	user 0.028s	sys 0.005s
I20240502 20:46:29.156982 30349 raft_consensus.cc:348] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.157310 30349 raft_consensus.cc:729] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:29.157487 30349 consensus_queue.cc:260] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.158815 30349 ts_tablet_manager.cc:1398] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:46:29.162148 27225 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.149749 30465 ts_tablet_manager.cc:902] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:29.158385 30295 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.166738 30295 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:29.167232 30465 tablet_copy_client.cc:308] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:29.167021 28847 raft_consensus.cc:1203] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:29.167503 30295 ts_tablet_manager.cc:1367] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.180s	user 0.033s	sys 0.003s
I20240502 20:46:29.167529 26812 catalog_manager.cc:5484] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.168210 27381 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 843a26fc48844bfd82911b6a27833023 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:53843)
I20240502 20:46:29.168208 30295 raft_consensus.cc:348] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:29.168382 27381 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 843a26fc48844bfd82911b6a27833023 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:53843: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-843a26fc48844bfd82911b6a27833023
I20240502 20:46:29.168406 30295 raft_consensus.cc:729] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:29.168562 30295 consensus_queue.cc:260] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:29.169945 27381 tablet_copy_source_session.cc:205] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.171615 30465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843a26fc48844bfd82911b6a27833023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.172595 30295 ts_tablet_manager.cc:1398] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 20:46:29.175158 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-d44f271f166b4c288b76689a3e3764d3 received from {username='slave'} at 127.19.66.134:40783
I20240502 20:46:29.175436 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-d44f271f166b4c288b76689a3e3764d3 on tablet d44f271f166b4c288b76689a3e3764d3 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:29.175945 27360 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:29.176548 30465 tablet_copy_client.cc:783] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.176960 30465 tablet_copy_client.cc:647] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.177508 30427 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:29.178241 30427 ts_tablet_manager.cc:1367] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.379s	user 0.033s	sys 0.002s
I20240502 20:46:29.179033 30427 raft_consensus.cc:348] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.179347 30427 raft_consensus.cc:729] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:29.179553 30427 consensus_queue.cc:260] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.180948 30427 ts_tablet_manager.cc:1398] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:46:29.181668 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-078a1591d9d74e31a0407aeab76911b4 received from {username='slave'} at 127.19.66.134:39121
I20240502 20:46:29.181778 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-078a1591d9d74e31a0407aeab76911b4 on tablet 078a1591d9d74e31a0407aeab76911b4 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:29.184070 26957 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.185272 30465 tablet_copy_client.cc:523] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.187446 30465 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:29.188999 26806 catalog_manager.cc:5484] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:29.192649 30384 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.193199 30384 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:29.193521 30384 ts_tablet_manager.cc:1367] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.631s	user 0.044s	sys 0.001s
I20240502 20:46:29.195113 28847 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:29.195503 27092 consensus_queue.cc:237] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
W20240502 20:46:29.198868 27026 consensus_peers.cc:485] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367543df62e048268908eb0d84c14876. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.199810 27360 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.200363 30194 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.204211 27225 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.171603 29251 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.207600 28847 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.211184 30194 raft_consensus.cc:1050] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:29.211239 30064 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.211889 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-a547561343304137a044ce11517f6007 received from {username='slave'} at 127.19.66.136:35339
I20240502 20:46:29.212007 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-a547561343304137a044ce11517f6007 on tablet a547561343304137a044ce11517f6007 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:29.211964 30194 consensus_queue.cc:237] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.212937 30193 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:46:29.213451 27024 consensus_peers.cc:485] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367543df62e048268908eb0d84c14876. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.216001 29251 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.216953 30064 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:46:29.217911 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 367543df62e048268908eb0d84c14876 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.219532 30384 raft_consensus.cc:348] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.221611 26957 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 20:46:29.222319 27024 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62ebc6576a4e4a108de55a4e9d7ff6b8. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.222541 30064 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.223058 30064 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.223587 30349 ts_tablet_manager.cc:902] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:29.224253 30349 tablet_copy_client.cc:308] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:29.224956 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet bf189c19164442e5af0827f1fb7dc085 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:35339)
I20240502 20:46:29.225246 30384 raft_consensus.cc:729] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:29.225507 30384 consensus_queue.cc:260] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.226480 30061 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.226665 28972 raft_consensus.cc:1203] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:29.226971 30384 ts_tablet_manager.cc:1398] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.001s
I20240502 20:46:29.226929 27360 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.232546 26805 catalog_manager.cc:5484] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.235165 26957 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.236349 30513 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.236726 30513 tablet_bootstrap.cc:492] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:29.236640 29251 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.236979 30513 ts_tablet_manager.cc:1367] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.203s	user 0.032s	sys 0.003s
I20240502 20:46:29.238113 30513 raft_consensus.cc:348] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:29.238299 30513 raft_consensus.cc:729] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:29.238962 26806 catalog_manager.cc:5484] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:29.241581 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-f00c4b7954564820875a25da5834416a received from {username='slave'} at 127.19.66.133:48853
I20240502 20:46:29.241731 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-f00c4b7954564820875a25da5834416a on tablet f00c4b7954564820875a25da5834416a with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:29.242439 28847 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.250650 26957 consensus_queue.cc:237] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:29.255585 30285 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.255786 27360 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:29.255681 30187 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.256374 30413 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:29.257193 30285 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:29.257560 30285 ts_tablet_manager.cc:1367] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.246s	user 0.033s	sys 0.007s
I20240502 20:46:29.257341 27362 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.258162 30285 raft_consensus.cc:348] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.258301 30285 raft_consensus.cc:729] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:29.258416 30285 consensus_queue.cc:260] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.258809 30285 ts_tablet_manager.cc:1398] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:29.259018 30513 consensus_queue.cc:260] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:29.259516 30513 ts_tablet_manager.cc:1398] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.022s	user 0.001s	sys 0.000s
I20240502 20:46:29.260942 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-d44f271f166b4c288b76689a3e3764d3 received from {username='slave'} at 127.19.66.133:48853
I20240502 20:46:29.260978 26980 tablet_copy_service.cc:342] P 297de823852d4a8da44791ab03131cd3: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-af693f5816e046f592fbd9b2e7ef0ad9 received from {username='slave'} at 127.19.66.136:35489
I20240502 20:46:29.261122 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-d44f271f166b4c288b76689a3e3764d3 on tablet d44f271f166b4c288b76689a3e3764d3 with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:29.261149 26980 tablet_copy_service.cc:433] P 297de823852d4a8da44791ab03131cd3: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-af693f5816e046f592fbd9b2e7ef0ad9 on tablet af693f5816e046f592fbd9b2e7ef0ad9 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:29.265376 30058 raft_consensus.cc:1050] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:29.266069 30058 consensus_queue.cc:237] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:29.266980 26805 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.269241 29116 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.270359 26797 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 367543df62e048268908eb0d84c14876 with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240502 20:46:29.270666 30187 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.270943 29251 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.271788 30061 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.276201 26957 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.276106 29252 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.277711 30058 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.281602 27360 consensus_queue.cc:237] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.282045 30501 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.282550 30501 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:29.282884 30501 ts_tablet_manager.cc:1367] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.458s	user 0.028s	sys 0.006s
I20240502 20:46:29.283617 30501 raft_consensus.cc:348] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.283875 30501 raft_consensus.cc:729] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:29.284054 30501 consensus_queue.cc:260] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.284619 30501 ts_tablet_manager.cc:1398] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
W20240502 20:46:29.284788 27297 consensus_peers.cc:485] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8deaa1f751034c15acd6ccaad37536c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.285228 28972 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.285281 27248 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-078a1591d9d74e31a0407aeab76911b4 received from {username='slave'} at 127.19.66.133:35399
I20240502 20:46:29.285373 27248 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-078a1591d9d74e31a0407aeab76911b4 on tablet 078a1591d9d74e31a0407aeab76911b4 with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:29.285702 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet bf189c19164442e5af0827f1fb7dc085 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:35339: session id = 6706d1c0946d42a9bae62d719e1e5c31-bf189c19164442e5af0827f1fb7dc085
I20240502 20:46:29.285750 27225 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.285831 27226 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:29.285948 27222 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.286499 29570 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.286603 26960 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.286576 30673 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:29.286859 27249 tablet_copy_source_session.cc:205] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.286988 29570 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.287319 27224 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.289158 27223 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.289934 29570 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.290697 30194 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.290992 30349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf189c19164442e5af0827f1fb7dc085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.291174 27092 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:29.291290 30194 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.293538 30349 tablet_copy_client.cc:783] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.293717 30349 tablet_copy_client.cc:647] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.294329 27223 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.295176 30349 tablet_copy_client.cc:523] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.296793 29252 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.296844 30349 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:29.298118 30284 raft_consensus.cc:1013] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: attempt to promote peer 1af1e87d6ff2448d9502de9d7c2b8ef3: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 20:46:29.298053 30061 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.298255 30284 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.298790 30673 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:29.299328 27224 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.302311 27226 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:29.304260 27361 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:29.307255 29117 raft_consensus.cc:1203] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:29.307986 27362 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.308311 30579 raft_consensus.cc:1050] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 to VOTER
I20240502 20:46:29.308946 30579 consensus_queue.cc:237] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:29.309630 30680 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.311697 28972 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.311964 27090 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.312732 30583 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.312808 29252 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.313397 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1b9ff3c921d474fa5ad106d1c17ab00 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.314144 30580 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.315325 30058 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.315436 27362 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.315798 27092 consensus_queue.cc:237] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.316097 30580 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.316751 27361 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.317986 26801 catalog_manager.cc:5484] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:29.318439 27359 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 20:46:29.319136 27026 consensus_peers.cc:485] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367543df62e048268908eb0d84c14876. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.319209 28847 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.319303 29324 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 20:46:29.320463 27024 consensus_peers.cc:485] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367543df62e048268908eb0d84c14876. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.320746 30194 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:29.321043 29324 raft_consensus.cc:1013] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: attempt to promote peer 1bc8a142cfc7487bacc573afb0a92b74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 20:46:29.322971 27223 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.324270 30064 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.325337 28847 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.325884 27092 consensus_queue.cc:237] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.328750 29116 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.329272 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 367543df62e048268908eb0d84c14876 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.329607 26805 catalog_manager.cc:5484] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 4 to 5, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:29.329787 30285 ts_tablet_manager.cc:902] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:29.330368 30285 tablet_copy_client.cc:308] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:29.331157 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet b5f81e46ab1f48fc8a88495b42f83640 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:35339)
I20240502 20:46:29.331175 30194 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.331279 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet b5f81e46ab1f48fc8a88495b42f83640 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:35339: session id = 6706d1c0946d42a9bae62d719e1e5c31-b5f81e46ab1f48fc8a88495b42f83640
I20240502 20:46:29.331580 27359 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.332463 27249 tablet_copy_source_session.cc:205] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.333673 30285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f81e46ab1f48fc8a88495b42f83640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.334067 30680 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.334558 26807 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:29.334940 29117 raft_consensus.cc:1203] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:29.336123 26959 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.337203 30194 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.338770 30285 tablet_copy_client.cc:783] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.339000 30285 tablet_copy_client.cc:647] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.338819 27362 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
W20240502 20:46:29.339776 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5704bdfe6c02437c8e9c8c3deaadb9e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.346246 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet d1b9ff3c921d474fa5ad106d1c17ab00 with delete_type TABLET_DATA_TOMBSTONED (TS c107c43e538d48f58625c4320f0aa4b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43554
I20240502 20:46:29.347573 29117 raft_consensus.cc:1203] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:29.348629 28980 raft_consensus.cc:1203] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:29.350898 26807 catalog_manager.cc:5484] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 7 to 8, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:29.355583 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet 367543df62e048268908eb0d84c14876 with delete_type TABLET_DATA_TOMBSTONED (TS c107c43e538d48f58625c4320f0aa4b8 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43554
I20240502 20:46:29.355942 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:29.357182 30187 raft_consensus.cc:1013] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: attempt to promote peer 1bc8a142cfc7487bacc573afb0a92b74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 20:46:29.363554 30061 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.366969 29251 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.367631 29116 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.369861 27227 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.371960 30061 raft_consensus.cc:1050] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:29.372588 30061 consensus_queue.cc:237] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:29.375901 29116 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.376010 29251 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.376366 30193 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.376529 30058 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.377703 26805 catalog_manager.cc:5484] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.378494 26957 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.379683 30061 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.380579 26957 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.382390 28847 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.383018 27227 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.383503 30061 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.387907 30061 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.387995 27227 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.391269 26957 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.391427 29251 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.393996 29116 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.396508 30285 tablet_copy_client.cc:523] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 20:46:29.398329 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2b61909194d488682afd91c0f796405. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.398514 30285 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:29.400225 26805 catalog_manager.cc:5484] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 6 to 7, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.400532 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:29.403541 26960 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.412019 28827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:29.412472 27093 consensus_queue.cc:237] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:29.413663 30284 raft_consensus.cc:1050] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:29.414271 30284 consensus_queue.cc:237] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:29.414762 29116 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.415254 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:29.415460 30058 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:29.416322 29117 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.416558 26960 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.417114 29570 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.417532 29570 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.421802 27093 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.421185 28972 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.422147 29251 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.423362 30058 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:29.424048 30284 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.419596 30582 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.427290 26957 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.428617 27092 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.428782 27360 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.430989 30058 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:29.433126 26807 catalog_manager.cc:5484] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.440374 30058 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.440279 28972 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.440666 27227 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.442934 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 135a08f1f2ac4707bbacee81ac73a19e with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.444192 29116 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.446667 29251 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.450328 26805 catalog_manager.cc:5484] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 7 to 8, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:46:29.451493 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84484d65e2bb4e23ac621a39434ce382. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.457163 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet 135a08f1f2ac4707bbacee81ac73a19e with delete_type TABLET_DATA_TOMBSTONED (TS c107c43e538d48f58625c4320f0aa4b8 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43554
I20240502 20:46:29.457268 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:29.457643 27093 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:29.458465 26957 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:29.458621 27093 consensus_queue.cc:237] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:29.459133 30580 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:29.459610 30580 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:29.459744 26959 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.460386 29252 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:29.463505 30187 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:29.464838 29116 raft_consensus.cc:1261] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:29.466044 30187 consensus_queue.cc:1035] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:29.467530 30437 ts_tablet_manager.cc:902] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:29.468045 30437 tablet_copy_client.cc:308] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:29.468508 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet 078a1591d9d74e31a0407aeab76911b4 from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:49537)
I20240502 20:46:29.468611 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet 078a1591d9d74e31a0407aeab76911b4 from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:49537: session id = 1bc8a142cfc7487bacc573afb0a92b74-078a1591d9d74e31a0407aeab76911b4
I20240502 20:46:29.469616 27249 tablet_copy_source_session.cc:205] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.470500 30443 ts_tablet_manager.cc:902] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:29.470338 30058 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.471107 30443 tablet_copy_client.cc:308] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:29.471742 30437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 078a1591d9d74e31a0407aeab76911b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.472592 29252 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.474514 30437 tablet_copy_client.cc:783] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.474676 30437 tablet_copy_client.cc:647] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.475445 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 135a08f1f2ac4707bbacee81ac73a19e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.475689 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet f00c4b7954564820875a25da5834416a from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:52059)
I20240502 20:46:29.475800 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet f00c4b7954564820875a25da5834416a from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:52059: session id = 1bc8a142cfc7487bacc573afb0a92b74-f00c4b7954564820875a25da5834416a
I20240502 20:46:29.477022 27112 tablet_copy_source_session.cc:205] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.478408 27226 consensus_queue.cc:237] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:29.480276 29116 raft_consensus.cc:2932] T 135a08f1f2ac4707bbacee81ac73a19e P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.480873 30443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00c4b7954564820875a25da5834416a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.481650 26802 catalog_manager.cc:5484] T 135a08f1f2ac4707bbacee81ac73a19e P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.483032 28972 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.483526 30443 tablet_copy_client.cc:783] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.483696 30443 tablet_copy_client.cc:647] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.483825 30581 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.484789 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet 135a08f1f2ac4707bbacee81ac73a19e with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:29.485514 27093 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.486439 30740 tablet_replica.cc:330] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: stopping tablet replica
I20240502 20:46:29.486845 30740 raft_consensus.cc:2229] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:29.487092 30740 raft_consensus.cc:2258] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:29.487668 30764 tablet_replica.cc:330] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:29.488493 30061 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.489369 30583 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.492833 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1545142b0bd40859bf33bf8686002ae with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.493999 28972 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.494900 30740 ts_tablet_manager.cc:1875] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:29.495486 30583 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.496331 27361 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.496492 29116 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.498633 26812 catalog_manager.cc:5484] T f1545142b0bd40859bf33bf8686002ae P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 5 to 6, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.498881 30764 raft_consensus.cc:2229] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:29.499253 30764 raft_consensus.cc:2258] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:29.499150 28847 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.500741 26957 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.501186 27093 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.502164 30583 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.504506 30680 raft_consensus.cc:1050] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:29.505515 30680 consensus_queue.cc:237] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.505913 26801 catalog_manager.cc:5484] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.506143 27093 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.508667 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet f1545142b0bd40859bf33bf8686002ae with delete_type TABLET_DATA_TOMBSTONED (TS ebd8300729ec4226acc38b9cff2d8d3b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45924
I20240502 20:46:29.510466 27361 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.510959 30061 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.511174 29115 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.511291 30489 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.511677 30489 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:29.511879 30489 ts_tablet_manager.cc:1367] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.417s	user 0.029s	sys 0.001s
I20240502 20:46:29.512419 30061 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.516788 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:29.517424 28848 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.517974 30187 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.518368 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:29.519502 30745 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.521232 29115 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.521776 26957 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.522568 27092 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.523242 27360 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.524839 26807 catalog_manager.cc:5484] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 5 to 6, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.527191 26957 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.528481 26805 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.534428 30771 tablet_replica.cc:330] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b: stopping tablet replica
I20240502 20:46:29.535022 30680 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:46:29.535388 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5704bdfe6c02437c8e9c8c3deaadb9e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.535912 26805 catalog_manager.cc:5484] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.536588 29117 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.539162 30437 tablet_copy_client.cc:523] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.540045 30465 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.540349 30465 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:29.540524 30465 ts_tablet_manager.cc:1367] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.353s	user 0.028s	sys 0.001s
I20240502 20:46:29.540916 30437 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:29.541846 26957 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.542255 30764 ts_tablet_manager.cc:1875] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:29.544891 30764 ts_tablet_manager.cc:1888] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:46:29.545004 30764 log.cc:1196] T 135a08f1f2ac4707bbacee81ac73a19e P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/135a08f1f2ac4707bbacee81ac73a19e
I20240502 20:46:29.550927 27227 consensus_queue.cc:237] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.551437 30771 raft_consensus.cc:2229] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:29.551800 30581 raft_consensus.cc:1050] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 to VOTER
I20240502 20:46:29.552219 30581 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } }
I20240502 20:46:29.552368 30771 raft_consensus.cc:2258] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:29.554169 28846 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.554981 30228 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:29.555569 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:29.555965 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5f81e46ab1f48fc8a88495b42f83640. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.557605 30443 tablet_copy_client.cc:523] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.559446 30443 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:29.561407 27361 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.562052 30228 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.564134 30740 ts_tablet_manager.cc:1888] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:29.564257 30740 log.cc:1196] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/d1b9ff3c921d474fa5ad106d1c17ab00
I20240502 20:46:29.565227 26797 catalog_manager.cc:4830] TS c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): tablet d1b9ff3c921d474fa5ad106d1c17ab00 (table rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]) successfully deleted
I20240502 20:46:29.566191 27360 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.568882 28847 raft_consensus.cc:1203] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
W20240502 20:46:29.569105 30349 log.cc:924] Time spent T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: Append to log took a long time: real 0.053s	user 0.003s	sys 0.000s
I20240502 20:46:29.571164 27093 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.571396 30771 ts_tablet_manager.cc:1875] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:29.575258 30771 ts_tablet_manager.cc:1888] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:29.512452 30489 raft_consensus.cc:348] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:29.578552 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet 135a08f1f2ac4707bbacee81ac73a19e (table rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]) successfully deleted
I20240502 20:46:29.579134 26957 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.580039 30759 ts_tablet_manager.cc:902] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:29.580992 30759 tablet_copy_client.cc:308] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:29.581595 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet fdd2311ec0b645f5a79098a32c6deba2 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:58553)
I20240502 20:46:29.581718 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet fdd2311ec0b645f5a79098a32c6deba2 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:58553: session id = 6706d1c0946d42a9bae62d719e1e5c31-fdd2311ec0b645f5a79098a32c6deba2
I20240502 20:46:29.582785 27112 tablet_copy_source_session.cc:205] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.583492 26959 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.573319 30580 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20240502 20:46:29.566712 30583 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20240502 20:46:29.586898 30228 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.565120 30740 tablet_replica.cc:330] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8: stopping tablet replica
I20240502 20:46:29.588714 30228 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.588973 30740 raft_consensus.cc:2229] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:29.589212 30740 raft_consensus.cc:2258] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:29.589517 26959 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.541051 30465 raft_consensus.cc:348] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:29.593271 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:29.593402 30465 raft_consensus.cc:729] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:29.593541 30465 consensus_queue.cc:260] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:29.577392 30489 raft_consensus.cc:729] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:29.570685 28972 raft_consensus.cc:1203] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 20:46:29.594125 30489 consensus_queue.cc:260] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:29.594497 30489 ts_tablet_manager.cc:1398] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.083s	user 0.003s	sys 0.000s
I20240502 20:46:29.594786 30465 ts_tablet_manager.cc:1398] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.054s	user 0.001s	sys 0.004s
I20240502 20:46:29.597209 30501 ts_tablet_manager.cc:902] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:29.598021 30501 tablet_copy_client.cc:308] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:29.599025 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:48853)
I20240502 20:46:29.599171 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:48853: session id = 1f40a11333e44ea0a34b6bc7618af522-62ebc6576a4e4a108de55a4e9d7ff6b8
I20240502 20:46:29.600539 27112 tablet_copy_source_session.cc:205] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.601743 30501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.603299 26810 catalog_manager.cc:5484] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:29.604533 30501 tablet_copy_client.cc:783] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.604779 30501 tablet_copy_client.cc:647] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.607498 28972 raft_consensus.cc:1203] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:29.608270 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-fdd2311ec0b645f5a79098a32c6deba2 received from {username='slave'} at 127.19.66.134:40783
I20240502 20:46:29.608393 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-fdd2311ec0b645f5a79098a32c6deba2 on tablet fdd2311ec0b645f5a79098a32c6deba2 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:29.611014 30513 ts_tablet_manager.cc:902] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:29.611594 30513 tablet_copy_client.cc:308] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:29.617486 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet fdd2311ec0b645f5a79098a32c6deba2 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:48853)
I20240502 20:46:29.617646 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet fdd2311ec0b645f5a79098a32c6deba2 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:48853: session id = 1f40a11333e44ea0a34b6bc7618af522-fdd2311ec0b645f5a79098a32c6deba2
I20240502 20:46:29.618959 27112 tablet_copy_source_session.cc:205] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.621506 30513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd2311ec0b645f5a79098a32c6deba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.622681 30740 ts_tablet_manager.cc:1875] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:29.626703 30740 ts_tablet_manager.cc:1888] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 20:46:29.626858 30740 log.cc:1196] T 367543df62e048268908eb0d84c14876 P c107c43e538d48f58625c4320f0aa4b8: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/367543df62e048268908eb0d84c14876
I20240502 20:46:29.627825 30740 tablet_replica.cc:330] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8: stopping tablet replica
I20240502 20:46:29.628103 30740 raft_consensus.cc:2229] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:29.627039 30513 tablet_copy_client.cc:783] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.628371 26797 catalog_manager.cc:4830] TS c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): tablet 367543df62e048268908eb0d84c14876 (table rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]) successfully deleted
I20240502 20:46:29.628556 30805 ts_tablet_manager.cc:902] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:29.628659 30740 raft_consensus.cc:2258] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:29.628856 30513 tablet_copy_client.cc:647] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.629711 30443 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.629864 30501 tablet_copy_client.cc:523] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.630373 30443 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:29.630649 30443 ts_tablet_manager.cc:1367] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.071s	user 0.034s	sys 0.000s
I20240502 20:46:29.631119 30513 tablet_copy_client.cc:523] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.631435 30443 raft_consensus.cc:348] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.631610 30443 raft_consensus.cc:729] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:29.631748 30443 consensus_queue.cc:260] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.631928 30501 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:29.632790 30513 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:29.633327 30583 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:29.636845 28845 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:29.638265 30805 tablet_copy_client.cc:308] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:29.639081 30443 ts_tablet_manager.cc:1398] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 20:46:29.641057 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-f00c4b7954564820875a25da5834416a received from {username='slave'} at 127.19.66.135:52059
I20240502 20:46:29.641216 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-f00c4b7954564820875a25da5834416a on tablet f00c4b7954564820875a25da5834416a with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:29.641728 26806 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 4 to 5, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:29.642237 30759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd2311ec0b645f5a79098a32c6deba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.645480 30759 tablet_copy_client.cc:783] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.645695 30759 tablet_copy_client.cc:647] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.647241 27226 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } }
I20240502 20:46:29.648923 28845 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.650303 30349 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.650633 30349 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:29.650837 30349 ts_tablet_manager.cc:1367] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.354s	user 0.031s	sys 0.000s
I20240502 20:46:29.651258 30027 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.651352 30349 raft_consensus.cc:348] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.651479 30349 raft_consensus.cc:729] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:29.651597 30349 consensus_queue.cc:260] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.652668 30349 ts_tablet_manager.cc:1398] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:29.653182 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-bf189c19164442e5af0827f1fb7dc085 received from {username='slave'} at 127.19.66.136:35339
I20240502 20:46:29.653273 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-bf189c19164442e5af0827f1fb7dc085 on tablet bf189c19164442e5af0827f1fb7dc085 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:29.657655 28972 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.657621 30027 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:29.663283 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a906569ab3b14520b4876bc8d5450f7d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.663846 28848 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:29.664717 30771 log.cc:1196] T f1545142b0bd40859bf33bf8686002ae P ebd8300729ec4226acc38b9cff2d8d3b: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/wals/f1545142b0bd40859bf33bf8686002ae
I20240502 20:46:29.665865 26796 catalog_manager.cc:4830] TS ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): tablet f1545142b0bd40859bf33bf8686002ae (table rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]) successfully deleted
I20240502 20:46:29.580469 27360 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.593353 27362 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.667321 30759 tablet_copy_client.cc:523] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.601384 27381 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-843a26fc48844bfd82911b6a27833023 received from {username='slave'} at 127.19.66.134:53843
I20240502 20:46:29.668121 27381 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-843a26fc48844bfd82911b6a27833023 on tablet 843a26fc48844bfd82911b6a27833023 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:29.637568 27361 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:29.671768 29117 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.672336 26807 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:29.674217 30759 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:29.674458 27092 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.674491 27226 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.674964 30712 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.675467 30712 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.676923 29570 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.677315 30147 raft_consensus.cc:1050] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 to VOTER
I20240502 20:46:29.677764 30147 consensus_queue.cc:237] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:29.679904 27359 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.679940 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet a906569ab3b14520b4876bc8d5450f7d with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47758
I20240502 20:46:29.680979 30248 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.681109 30712 raft_consensus.cc:1013] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: attempt to promote peer 1bc8a142cfc7487bacc573afb0a92b74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 20:46:29.682410 28980 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.682546 30825 tablet_replica.cc:330] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:46:29.683158 26959 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.683259 30825 raft_consensus.cc:2229] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:29.684095 29252 raft_consensus.cc:1203] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 20:46:29.685272 27092 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.685717 30284 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.686213 30284 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.687646 30284 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.687471 29570 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.683516 30825 pending_rounds.cc:62] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Trying to abort 1 pending ops.
I20240502 20:46:29.688130 30825 pending_rounds.cc:68] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023340706004774912 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "a906569ab3b14520b4876bc8d5450f7d" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } } }
I20240502 20:46:29.688344 30825 raft_consensus.cc:2894] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 20:46:29.688413 30825 raft_consensus.cc:2866] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 20:46:29.688460 30825 raft_consensus.cc:2258] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:29.688714 27359 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:29.690045 29117 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.693270 26802 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.698611 30712 raft_consensus.cc:1050] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:29.698828 27226 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.699090 30712 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.701041 29117 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.702106 30284 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.703272 27226 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.703837 30284 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.707885 30740 ts_tablet_manager.cc:1875] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 20:46:29.709419 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cf737e20ca74db1b94ccff5d60133ee. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.710796 30443 ts_tablet_manager.cc:902] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:29.711416 30443 tablet_copy_client.cc:308] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:29.712852 28980 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.713829 27381 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 8deaa1f751034c15acd6ccaad37536c6 from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:58959)
I20240502 20:46:29.713953 27381 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 8deaa1f751034c15acd6ccaad37536c6 from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:58959: session id = 1bc8a142cfc7487bacc573afb0a92b74-8deaa1f751034c15acd6ccaad37536c6
I20240502 20:46:29.714995 27381 tablet_copy_source_session.cc:205] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.715648 30147 raft_consensus.cc:1050] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 to VOTER
I20240502 20:46:29.715690 30740 ts_tablet_manager.cc:1888] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 20:46:29.715771 30740 log.cc:1196] T 135a08f1f2ac4707bbacee81ac73a19e P c107c43e538d48f58625c4320f0aa4b8: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/135a08f1f2ac4707bbacee81ac73a19e
I20240502 20:46:29.716193 30147 consensus_queue.cc:237] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.716573 30443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8deaa1f751034c15acd6ccaad37536c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.716517 26797 catalog_manager.cc:4830] TS c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): tablet 135a08f1f2ac4707bbacee81ac73a19e (table rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]) successfully deleted
W20240502 20:46:29.719098 27297 consensus_peers.cc:485] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8deaa1f751034c15acd6ccaad37536c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.720196 26957 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.720742 28980 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.720903 30443 tablet_copy_client.cc:783] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.721208 30443 tablet_copy_client.cc:647] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.721575 30712 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.721830 27227 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.721944 30712 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.722429 30712 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.724244 30285 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.724617 30285 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:29.724885 30285 ts_tablet_manager.cc:1367] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.326s	user 0.023s	sys 0.004s
I20240502 20:46:29.727646 29570 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.729132 30285 raft_consensus.cc:348] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.729336 30285 raft_consensus.cc:729] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:29.729475 30285 consensus_queue.cc:260] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.730886 30285 ts_tablet_manager.cc:1398] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 20:46:29.731424 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-b5f81e46ab1f48fc8a88495b42f83640 received from {username='slave'} at 127.19.66.136:35339
I20240502 20:46:29.731523 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-b5f81e46ab1f48fc8a88495b42f83640 on tablet b5f81e46ab1f48fc8a88495b42f83640 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:29.732048 26957 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.735697 28980 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.739035 26810 catalog_manager.cc:5484] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 5 to 6, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.739742 27227 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.740743 29570 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
W20240502 20:46:29.741312 30789 consensus_queue.cc:1159] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d1f3f8e444364d6faa42d651b5dbb997" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240502 20:46:29.742290 30443 tablet_copy_client.cc:523] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.743501 30825 ts_tablet_manager.cc:1875] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:29.744365 30443 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:29.744519 29117 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.747007 30825 ts_tablet_manager.cc:1888] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:29.747452 30825 log.cc:1196] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/a906569ab3b14520b4876bc8d5450f7d
I20240502 20:46:29.748288 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet a906569ab3b14520b4876bc8d5450f7d (table rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]) successfully deleted
I20240502 20:46:29.748999 30061 raft_consensus.cc:1050] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 to VOTER
I20240502 20:46:29.749567 30061 consensus_queue.cc:237] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:29.754410 26807 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 5 to 6, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.755630 27227 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.755878 27361 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.756670 30187 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.757884 26957 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.758455 30187 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.761617 30850 ts_tablet_manager.cc:902] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:29.762282 28980 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.764142 30187 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.764670 27381 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 5586b9e60b544aed87c24f7b3ce0198c from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:44893)
I20240502 20:46:29.764816 27381 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 5586b9e60b544aed87c24f7b3ce0198c from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:44893: session id = 6706d1c0946d42a9bae62d719e1e5c31-5586b9e60b544aed87c24f7b3ce0198c
I20240502 20:46:29.766010 27381 tablet_copy_source_session.cc:205] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.766710 30248 raft_consensus.cc:1050] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:29.766909 29252 raft_consensus.cc:1203] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:29.767165 30248 consensus_queue.cc:237] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:29.767810 27362 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:29.769275 29116 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.769742 30582 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.770619 29116 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.770656 26957 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.771090 30027 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.771603 27093 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.771868 30147 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.772881 30582 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.774266 27359 consensus_queue.cc:237] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.776199 28974 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.776700 30284 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.776888 30850 tablet_copy_client.cc:308] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:29.777575 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet 367543df62e048268908eb0d84c14876 from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:52059)
I20240502 20:46:29.777704 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet 367543df62e048268908eb0d84c14876 from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:52059: session id = 1bc8a142cfc7487bacc573afb0a92b74-367543df62e048268908eb0d84c14876
I20240502 20:46:29.778985 27112 tablet_copy_source_session.cc:205] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 20:46:29.779500 27297 consensus_peers.cc:485] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.779884 30501 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.780221 30501 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:29.780315 30147 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.780679 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367543df62e048268908eb0d84c14876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.780543 30501 ts_tablet_manager.cc:1367] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.149s	user 0.031s	sys 0.005s
I20240502 20:46:29.782336 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8deaa1f751034c15acd6ccaad37536c6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.782428 30501 raft_consensus.cc:348] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.784817 28980 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.785851 29252 raft_consensus.cc:1203] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:29.785254 30437 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:29.787700 30805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5586b9e60b544aed87c24f7b3ce0198c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.787741 30437 tablet_bootstrap.cc:492] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:29.787994 30437 ts_tablet_manager.cc:1367] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.247s	user 0.035s	sys 0.003s
I20240502 20:46:29.788653 30437 raft_consensus.cc:348] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.789180 30437 raft_consensus.cc:729] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:29.789338 30437 consensus_queue.cc:260] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.789335 30187 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.790500 30805 tablet_copy_client.cc:783] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.790530 30437 ts_tablet_manager.cc:1398] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:46:29.790854 30805 tablet_copy_client.cc:647] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.791404 27226 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.791981 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-078a1591d9d74e31a0407aeab76911b4 received from {username='slave'} at 127.19.66.135:49537
I20240502 20:46:29.792073 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-078a1591d9d74e31a0407aeab76911b4 on tablet 078a1591d9d74e31a0407aeab76911b4 with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:29.792721 30284 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.793112 26807 catalog_manager.cc:5484] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 6 to 7, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:29.793308 27361 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.793809 27227 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.795565 26957 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.796325 30805 tablet_copy_client.cc:523] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.796309 27089 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.796772 27091 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.797677 28980 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.798468 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet 8deaa1f751034c15acd6ccaad37536c6 with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:29.798753 30764 tablet_replica.cc:330] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:29.798909 30764 raft_consensus.cc:2229] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:29.799121 30764 raft_consensus.cc:2258] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:29.799103 26810 catalog_manager.cc:5484] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:29.800079 30764 ts_tablet_manager.cc:1875] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:29.801193 30284 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.802937 30764 ts_tablet_manager.cc:1888] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:46:29.803020 30764 log.cc:1196] T 8deaa1f751034c15acd6ccaad37536c6 P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/8deaa1f751034c15acd6ccaad37536c6
I20240502 20:46:29.803603 30805 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:29.803642 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet 8deaa1f751034c15acd6ccaad37536c6 (table rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]) successfully deleted
I20240502 20:46:29.805399 27091 consensus_queue.cc:237] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:29.805297 26957 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.805824 28980 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.807399 30501 raft_consensus.cc:729] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:29.807427 28972 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.807585 30501 consensus_queue.cc:260] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:29.808465 30193 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.809706 30501 ts_tablet_manager.cc:1398] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.028s	user 0.001s	sys 0.000s
I20240502 20:46:29.810161 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-62ebc6576a4e4a108de55a4e9d7ff6b8 received from {username='slave'} at 127.19.66.133:48853
I20240502 20:46:29.810236 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-62ebc6576a4e4a108de55a4e9d7ff6b8 on tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:29.812779 30850 tablet_copy_client.cc:783] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.813511 30850 tablet_copy_client.cc:647] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.815027 26810 catalog_manager.cc:5484] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.818367 30061 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.821219 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.820955 30027 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.822167 28972 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.825518 29116 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.825966 26802 catalog_manager.cc:5484] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.829403 27361 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.829675 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9fef57f6eb490da536170a02b96212 with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:29.829998 27227 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.830221 30764 tablet_replica.cc:330] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:29.830402 30764 raft_consensus.cc:2229] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:29.830590 30764 raft_consensus.cc:2258] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:29.830682 26807 catalog_manager.cc:5484] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:29.831398 30764 ts_tablet_manager.cc:1875] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:29.831476 29570 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.831874 29570 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.832690 30193 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.833689 26957 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.834261 30764 ts_tablet_manager.cc:1888] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:29.834329 30764 log.cc:1196] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/ff9fef57f6eb490da536170a02b96212
I20240502 20:46:29.834175 27093 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.835330 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet ff9fef57f6eb490da536170a02b96212 (table rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]) successfully deleted
I20240502 20:46:29.835698 30248 raft_consensus.cc:1050] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 to VOTER
I20240502 20:46:29.836236 30248 consensus_queue.cc:237] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.838236 28848 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.838236 26959 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.838467 28972 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.838439 27093 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.838939 30582 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.839906 30580 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.840201 30580 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.840538 27093 raft_consensus.cc:858] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
W20240502 20:46:29.840622 27093 raft_consensus.cc:1565] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:29.841007 27093 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.842460 27361 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.842933 29116 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.843067 30580 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.844139 30580 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.844615 30580 raft_consensus.cc:1013] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: attempt to promote peer 1f40a11333e44ea0a34b6bc7618af522: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 20:46:29.845297 30583 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.846877 27359 consensus_queue.cc:237] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:29.848429 27359 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.848594 28848 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.850875 26959 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.851487 27227 consensus_queue.cc:237] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:29.853307 30147 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.853312 28972 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.853536 29115 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.855062 30581 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.855358 28980 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.855876 30831 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.856055 30579 raft_consensus.cc:1050] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 to VOTER
I20240502 20:46:29.856709 30579 consensus_queue.cc:237] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:29.848202 26957 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.858734 27091 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.860030 26960 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.860828 30465 ts_tablet_manager.cc:902] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:29.860803 27359 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.861717 30465 tablet_copy_client.cc:308] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:38835
I20240502 20:46:29.863654 26960 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.864163 30580 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.866291 30027 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:29.867933 30489 ts_tablet_manager.cc:902] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:29.868563 30489 tablet_copy_client.cc:308] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:29.869060 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet 367543df62e048268908eb0d84c14876 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:40783)
I20240502 20:46:29.869252 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet 367543df62e048268908eb0d84c14876 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:40783: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-367543df62e048268908eb0d84c14876
I20240502 20:46:29.869609 28848 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.870194 30583 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:29.870481 27112 tablet_copy_source_session.cc:205] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.871987 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367543df62e048268908eb0d84c14876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.872375 27359 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.872833 30580 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:29.872751 30583 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.872897 30581 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.874012 27361 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.874835 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b466a750cc6b4382947282d8aa6aeb1f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.875120 30489 tablet_copy_client.cc:783] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.875335 30489 tablet_copy_client.cc:647] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.876106 26960 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.878629 30745 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.879303 26806 catalog_manager.cc:5484] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.880203 28848 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.881696 30712 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.882091 30284 raft_consensus.cc:1050] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:29.882488 30284 consensus_queue.cc:237] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:29.882673 28980 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.883281 30580 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:29.884001 29117 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.884630 29117 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.885354 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 14a7bead40fb4eb0bec6b00804b52595 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.885563 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5cf737e20ca74db1b94ccff5d60133ee with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.885722 30147 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.886898 26959 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.887205 28980 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.889264 30850 tablet_copy_client.cc:523] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.889343 27226 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.889768 27227 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.889976 26960 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:29.890105 26812 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.890540 30712 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.891209 29114 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.891474 30850 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:29.893360 30712 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:29.895160 29115 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.895040 27093 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.896027 30027 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.897312 29115 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.899042 29115 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:29.899235 28972 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.907505 26805 catalog_manager.cc:5484] T b466a750cc6b4382947282d8aa6aeb1f P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.907831 26959 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.909237 30582 raft_consensus.cc:1050] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:29.909787 30582 consensus_queue.cc:237] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:29.912569 29115 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:29.914016 30579 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:29.916319 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet b466a750cc6b4382947282d8aa6aeb1f with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47758
I20240502 20:46:29.916698 30583 raft_consensus.cc:1050] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 to VOTER
I20240502 20:46:29.917255 30825 tablet_replica.cc:330] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:46:29.917259 30583 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.917482 30825 raft_consensus.cc:2229] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:29.917713 30825 raft_consensus.cc:2258] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:29.918689 26956 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:29.918870 27359 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:29.919309 29252 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.919324 30580 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:29.919811 26959 consensus_queue.cc:237] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:29.921185 30579 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.922173 27091 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:29.922726 30413 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:29.922842 27226 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:29.923485 30413 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:29.924270 28972 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 20:46:29.924443 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2dfc0bbbbca4cc28a97480007343424. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:29.925177 30437 ts_tablet_manager.cc:902] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:29.925631 30825 ts_tablet_manager.cc:1875] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:29.925769 30437 tablet_copy_client.cc:308] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:29.926162 26807 catalog_manager.cc:5484] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:29.927050 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet 84484d65e2bb4e23ac621a39434ce382 from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:49537)
I20240502 20:46:29.927161 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet 84484d65e2bb4e23ac621a39434ce382 from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:49537: session id = 1bc8a142cfc7487bacc573afb0a92b74-84484d65e2bb4e23ac621a39434ce382
I20240502 20:46:29.928172 27249 tablet_copy_source_session.cc:205] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.928344 30825 ts_tablet_manager.cc:1888] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:29.928427 30825 log.cc:1196] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/b466a750cc6b4382947282d8aa6aeb1f
I20240502 20:46:29.929858 30248 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:29.930018 30437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84484d65e2bb4e23ac621a39434ce382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.930248 30248 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:29.930256 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet b466a750cc6b4382947282d8aa6aeb1f (table rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]) successfully deleted
I20240502 20:46:29.931648 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.932078 30580 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.932866 27089 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.933254 30580 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.934018 28845 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:29.934397 30580 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:29.937389 30248 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.938180 28848 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:29.939229 27093 consensus_queue.cc:237] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:29.940377 29115 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.941439 29251 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.941836 30583 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:29.942020 30064 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.942551 30437 tablet_copy_client.cc:783] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.942695 27359 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.942926 28972 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.945322 27223 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.945652 26957 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.946163 26959 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.946233 30064 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.946563 30058 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.948408 30058 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.949049 28848 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:29.950716 26806 catalog_manager.cc:5484] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:29.950693 29251 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.951313 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:29.956466 26980 tablet_copy_service.cc:140] P 297de823852d4a8da44791ab03131cd3: Received BeginTabletCopySession request for tablet a2b61909194d488682afd91c0f796405 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:52999)
I20240502 20:46:29.956620 26980 tablet_copy_service.cc:161] P 297de823852d4a8da44791ab03131cd3: Beginning new tablet copy session on tablet a2b61909194d488682afd91c0f796405 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:52999: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-a2b61909194d488682afd91c0f796405
I20240502 20:46:29.956460 26812 catalog_manager.cc:5484] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 8 to 9, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:29.956669 27223 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.957695 26980 tablet_copy_source_session.cc:205] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.958847 30465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b61909194d488682afd91c0f796405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.961216 30437 tablet_copy_client.cc:647] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.962164 30465 tablet_copy_client.cc:783] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.962019 26807 catalog_manager.cc:5484] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:29.962337 30465 tablet_copy_client.cc:647] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.963395 27223 consensus_queue.cc:237] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:29.964222 30465 tablet_copy_client.cc:523] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.964643 30489 tablet_copy_client.cc:523] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:29.966542 30489 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:29.967552 30465 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:29.967708 26959 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:29.968679 30248 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:29.969450 29250 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:29.970978 30413 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.971671 27359 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:29.973435 27091 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.974715 29114 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.975373 28845 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:29.975464 29570 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:29.975930 29115 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:29.975818 27226 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.978240 30579 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:29.978420 30228 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:29.979627 26957 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:29.981655 30501 ts_tablet_manager.cc:902] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:29.982267 30501 tablet_copy_client.cc:308] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:29.983155 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet b5f81e46ab1f48fc8a88495b42f83640 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:35399)
I20240502 20:46:29.983285 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet b5f81e46ab1f48fc8a88495b42f83640 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:35399: session id = 1f40a11333e44ea0a34b6bc7618af522-b5f81e46ab1f48fc8a88495b42f83640
I20240502 20:46:29.984347 27249 tablet_copy_source_session.cc:205] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:29.985291 26812 catalog_manager.cc:5484] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index -1 to 4, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.985713 30501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f81e46ab1f48fc8a88495b42f83640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:29.986162 30228 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.986663 27361 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.991441 30501 tablet_copy_client.cc:783] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:29.991667 30501 tablet_copy_client.cc:647] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:29.991897 30579 raft_consensus.cc:1050] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:29.992869 29252 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:29.994747 27091 consensus_queue.cc:237] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:29.995652 30579 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:29.997578 29250 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:29.997722 27226 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.998072 30581 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:29.998127 30061 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:29.999418 29252 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:29.999864 30061 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.000344 27091 consensus_queue.cc:237] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.000746 30194 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.001377 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.003304 27226 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.004231 29570 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.005092 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.006448 29116 raft_consensus.cc:1203] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 20:46:30.008240 29116 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
W20240502 20:46:30.009446 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5704bdfe6c02437c8e9c8c3deaadb9e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:30.009490 26802 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:30.010985 30583 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.011202 26807 catalog_manager.cc:5484] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 7 to 8, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:30.011704 29252 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.013545 28972 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:30.014555 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5cf737e20ca74db1b94ccff5d60133ee with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.014508 28845 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:30.016116 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:30.016319 29116 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.016382 30764 tablet_replica.cc:330] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:30.016548 30764 raft_consensus.cc:2229] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.016736 30764 raft_consensus.cc:2258] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.017132 30194 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.017616 30764 ts_tablet_manager.cc:1875] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.019796 30061 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.020081 30764 ts_tablet_manager.cc:1888] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 20:46:30.020154 30764 log.cc:1196] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/62ebc6576a4e4a108de55a4e9d7ff6b8
I20240502 20:46:30.020295 28845 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.020541 30580 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:30.022217 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 (table rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]) successfully deleted
I20240502 20:46:30.022697 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.022658 30583 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.023375 29114 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.024433 30228 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.029168 29250 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.030018 26812 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.030388 26812 catalog_manager.cc:5484] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 6 to 7, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.030841 30759 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:30.031183 30759 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:30.031374 30759 ts_tablet_manager.cc:1367] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.357s	user 0.033s	sys 0.000s
I20240502 20:46:30.031860 30759 raft_consensus.cc:348] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.032024 30759 raft_consensus.cc:729] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:30.032168 30759 consensus_queue.cc:260] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.032600 30759 ts_tablet_manager.cc:1398] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:46:30.033233 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 078a1591d9d74e31a0407aeab76911b4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.033720 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-fdd2311ec0b645f5a79098a32c6deba2 received from {username='slave'} at 127.19.66.136:58553
I20240502 20:46:30.033835 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-fdd2311ec0b645f5a79098a32c6deba2 on tablet fdd2311ec0b645f5a79098a32c6deba2 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:30.034989 26810 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 6 to 7, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:30.035207 27361 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.036211 26957 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.037673 30064 raft_consensus.cc:1050] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 to VOTER
I20240502 20:46:30.038229 30064 consensus_queue.cc:237] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:30.040243 27360 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.040858 28845 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.041334 30194 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.043489 26810 catalog_manager.cc:5484] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 9 to 10, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:30.044740 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf737e20ca74db1b94ccff5d60133ee with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47758
I20240502 20:46:30.044968 30825 tablet_replica.cc:330] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:46:30.045154 30825 raft_consensus.cc:2229] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.045377 30825 raft_consensus.cc:2258] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.045501 30580 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.045986 26957 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.046202 30825 ts_tablet_manager.cc:1875] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.046573 30194 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.047066 28972 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.047605 30194 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.048820 30825 ts_tablet_manager.cc:1888] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 20:46:30.048902 30825 log.cc:1196] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/5cf737e20ca74db1b94ccff5d60133ee
I20240502 20:46:30.050249 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:30.050506 30764 tablet_replica.cc:330] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:30.050655 30764 raft_consensus.cc:2229] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.050904 30764 raft_consensus.cc:2258] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.051682 30764 ts_tablet_manager.cc:1875] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.054450 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet 5cf737e20ca74db1b94ccff5d60133ee (table rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]) successfully deleted
I20240502 20:46:30.054949 30712 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.055096 29115 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.055467 30764 ts_tablet_manager.cc:1888] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:46:30.055557 30764 log.cc:1196] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/5704bdfe6c02437c8e9c8c3deaadb9e3
I20240502 20:46:30.056321 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet 078a1591d9d74e31a0407aeab76911b4 with delete_type TABLET_DATA_TOMBSTONED (TS ebd8300729ec4226acc38b9cff2d8d3b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45924
I20240502 20:46:30.056581 30771 tablet_replica.cc:330] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: stopping tablet replica
I20240502 20:46:30.056757 30771 raft_consensus.cc:2229] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.057054 30771 raft_consensus.cc:2258] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.057108 27227 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:30.058610 29250 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.058907 30771 ts_tablet_manager.cc:1875] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.059397 30228 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.060062 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.061703 26960 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.062337 27227 consensus_queue.cc:237] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:30.063233 30581 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.064738 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 (table rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]) successfully deleted
I20240502 20:46:30.065721 29115 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:46:30.066289 30771 ts_tablet_manager.cc:1888] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 20:46:30.066401 30771 log.cc:1196] T 078a1591d9d74e31a0407aeab76911b4 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/wals/078a1591d9d74e31a0407aeab76911b4
I20240502 20:46:30.067272 26796 catalog_manager.cc:4830] TS ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): tablet 078a1591d9d74e31a0407aeab76911b4 (table rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]) successfully deleted
I20240502 20:46:30.066917 30579 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240502 20:46:30.069427 29117 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.073550 30579 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.074086 26801 catalog_manager.cc:5484] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.076952 30194 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.077620 29250 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.077847 27361 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.080147 26810 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:30.080289 27360 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.083348 27359 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.084379 30437 tablet_copy_client.cc:523] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:30.086321 30437 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:30.086741 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bf189c19164442e5af0827f1fb7dc085 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.087729 27227 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:30.089233 29250 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:30.089622 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:30.089893 30582 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:30.090180 30582 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:30.090405 27070 tablet_service.cc:1514] Processing DeleteTablet for tablet bf189c19164442e5af0827f1fb7dc085 with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47758
I20240502 20:46:30.090482 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet 14a7bead40fb4eb0bec6b00804b52595 with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47758
I20240502 20:46:30.090703 30825 tablet_replica.cc:330] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:46:30.090844 30825 raft_consensus.cc:2229] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.091003 30825 pending_rounds.cc:62] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Trying to abort 1 pending ops.
I20240502 20:46:30.091032 30825 pending_rounds.cc:68] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023340707499401216 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "bf189c19164442e5af0827f1fb7dc085" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } } }
I20240502 20:46:30.091202 30825 raft_consensus.cc:2894] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240502 20:46:30.091264 30825 raft_consensus.cc:2866] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 20:46:30.091310 30825 raft_consensus.cc:2258] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.094177 30027 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.096863 29252 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.097440 27361 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
W20240502 20:46:30.097939 27158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.66.133:45949, user_credentials={real_user=slave}} blocked reactor thread for 51719.4us
I20240502 20:46:30.099251 28847 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.100291 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bf189c19164442e5af0827f1fb7dc085 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.101112 26807 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 8 to 9, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.104552 27227 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.109498 30284 raft_consensus.cc:1050] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 to VOTER
I20240502 20:46:30.109524 28847 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.109989 30284 consensus_queue.cc:237] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:30.115937 26960 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:30.116670 30712 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:30.119927 30443 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:30.120513 30443 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:30.120879 30443 ts_tablet_manager.cc:1367] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.377s	user 0.031s	sys 0.004s
I20240502 20:46:30.121630 30443 raft_consensus.cc:348] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:30.122035 30443 raft_consensus.cc:729] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:30.122407 30443 consensus_queue.cc:260] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:30.125600 28980 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:30.126149 30876 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:30.127202 30894 consensus_queue.cc:1159] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d1f3f8e444364d6faa42d651b5dbb997" status { last_received { term: 1 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 6 } }
I20240502 20:46:30.128715 30825 ts_tablet_manager.cc:1875] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.131431 30825 ts_tablet_manager.cc:1888] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:46:30.131515 30825 log.cc:1196] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/bf189c19164442e5af0827f1fb7dc085
I20240502 20:46:30.132042 30825 tablet_replica.cc:330] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:46:30.132122 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet bf189c19164442e5af0827f1fb7dc085 (table rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]) successfully deleted
I20240502 20:46:30.132223 30825 raft_consensus.cc:2229] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.132441 30825 raft_consensus.cc:2258] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.133221 28974 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:46:30.133350 30825 ts_tablet_manager.cc:1875] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.134135 28847 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:46:30.134871 30061 raft_consensus.cc:1050] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 to VOTER
I20240502 20:46:30.135381 30061 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:30.135425 30579 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:30.135946 30825 ts_tablet_manager.cc:1888] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:30.136021 30825 log.cc:1196] T 14a7bead40fb4eb0bec6b00804b52595 P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/14a7bead40fb4eb0bec6b00804b52595
I20240502 20:46:30.136632 28974 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.137553 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet 14a7bead40fb4eb0bec6b00804b52595 (table rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]) successfully deleted
I20240502 20:46:30.137895 26957 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.138484 30194 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.139317 28847 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.140007 30194 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.142764 30194 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.145891 30712 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.146978 30187 raft_consensus.cc:1050] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 to VOTER
I20240502 20:46:30.147634 30187 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.149845 28974 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.150259 26959 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.151003 29250 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.151567 27226 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.152141 30058 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.152330 30061 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.152590 30064 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.154210 30058 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:46:30.157102 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:30.157567 28848 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.137455 29115 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.159490 30194 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.160305 30443 ts_tablet_manager.cc:1398] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.039s	user 0.001s	sys 0.000s
I20240502 20:46:30.161702 27227 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:30.163074 27227 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.163568 28827 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf737e20ca74db1b94ccff5d60133ee with delete_type TABLET_DATA_TOMBSTONED (TS 1f40a11333e44ea0a34b6bc7618af522 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47924
I20240502 20:46:30.164120 28826 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 with delete_type TABLET_DATA_TOMBSTONED (TS 1f40a11333e44ea0a34b6bc7618af522 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47924
I20240502 20:46:30.164263 28825 tablet_service.cc:1514] Processing DeleteTablet for tablet bf189c19164442e5af0827f1fb7dc085 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 7 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:47924
I20240502 20:46:30.166005 30187 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:30.166998 27227 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.171011 26806 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:30.171108 26957 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:30.175660 27093 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } }
I20240502 20:46:30.176210 30680 raft_consensus.cc:1013] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: attempt to promote peer 1af1e87d6ff2448d9502de9d7c2b8ef3: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 20:46:30.176623 30579 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:46:30.177572 26957 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.178236 30680 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.178685 28847 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.178934 30583 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.179909 30680 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.179700 28848 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:30.182608 27361 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.185691 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 078a1591d9d74e31a0407aeab76911b4 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.186350 29117 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.186288 30680 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.188972 26957 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.190495 26806 catalog_manager.cc:5484] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 10 to 11, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:30.191525 30058 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:30.191944 26807 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:30.192575 28972 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.193935 28846 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:30.196676 26959 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:30.199467 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet 078a1591d9d74e31a0407aeab76911b4 with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:30.199858 30764 tablet_replica.cc:330] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:30.200050 30764 raft_consensus.cc:2229] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.200405 30764 raft_consensus.cc:2258] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.195133 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f00c4b7954564820875a25da5834416a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.202240 30764 ts_tablet_manager.cc:1875] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.203095 27227 consensus_queue.cc:237] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:30.206427 29117 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:46:30.207237 27093 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.207307 26812 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:30.208153 30764 ts_tablet_manager.cc:1888] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 20:46:30.208508 30764 log.cc:1196] T 078a1591d9d74e31a0407aeab76911b4 P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/078a1591d9d74e31a0407aeab76911b4
I20240502 20:46:30.209417 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet 078a1591d9d74e31a0407aeab76911b4 (table rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]) successfully deleted
I20240502 20:46:30.212291 27093 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } }
I20240502 20:46:30.212713 26957 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.212821 30583 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:46:30.214607 28846 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.214589 30187 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.215617 30187 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.215837 30058 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:30.216435 26957 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.217893 30194 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.217836 30027 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.221300 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 078a1591d9d74e31a0407aeab76911b4 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.223263 26807 catalog_manager.cc:5484] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 11 to 12, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:30.223644 28846 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:30.224009 30064 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:30.223956 29117 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.226157 28845 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:30.226613 26957 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:30.228359 28825 tablet_service.cc:1514] Processing DeleteTablet for tablet 078a1591d9d74e31a0407aeab76911b4 with delete_type TABLET_DATA_TOMBSTONED (TS 1f40a11333e44ea0a34b6bc7618af522 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:47924
I20240502 20:46:30.230779 26812 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
W20240502 20:46:30.231386 26794 catalog_manager.cc:4797] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): delete failed for tablet 5cf737e20ca74db1b94ccff5d60133ee because the tablet was not found. No further retry: Not found: Tablet not found: 5cf737e20ca74db1b94ccff5d60133ee
I20240502 20:46:30.231550 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f00c4b7954564820875a25da5834416a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.231796 30969 tablet_replica.cc:330] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: stopping tablet replica
I20240502 20:46:30.232373 30969 raft_consensus.cc:2229] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:30.232677 30969 raft_consensus.cc:2258] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:30.233845 27227 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.233994 26959 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:30.233968 28974 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:30.234390 30194 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.236337 27226 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.238373 26810 catalog_manager.cc:5484] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:30.238757 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet f00c4b7954564820875a25da5834416a with delete_type TABLET_DATA_TOMBSTONED (TS ebd8300729ec4226acc38b9cff2d8d3b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45924
I20240502 20:46:30.241125 30771 tablet_replica.cc:330] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: stopping tablet replica
I20240502 20:46:30.241369 30771 raft_consensus.cc:2229] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.241603 30771 raft_consensus.cc:2258] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 20:46:30.241598 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:30.242558 30771 ts_tablet_manager.cc:1875] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.242354 29324 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.242624 30187 raft_consensus.cc:1050] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 to VOTER
I20240502 20:46:30.242686 27226 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.243153 30187 consensus_queue.cc:237] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:30.247769 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fdd2311ec0b645f5a79098a32c6deba2 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.248819 30771 ts_tablet_manager.cc:1888] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:30.248927 30771 log.cc:1196] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/wals/f00c4b7954564820875a25da5834416a
I20240502 20:46:30.248867 26812 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 7 to 8, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:30.249831 26796 catalog_manager.cc:4830] TS ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): tablet f00c4b7954564820875a25da5834416a (table rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]) successfully deleted
I20240502 20:46:30.251569 28974 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.251637 30513 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:30.251591 26959 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.251989 30513 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:30.252229 30513 ts_tablet_manager.cc:1367] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.619s	user 0.033s	sys 0.000s
I20240502 20:46:30.252641 28846 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.252447 26959 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.254921 29324 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.255312 29324 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.258369 29324 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240502 20:46:30.259732 27381 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-8deaa1f751034c15acd6ccaad37536c6 received from {username='slave'} at 127.19.66.135:58959
I20240502 20:46:30.259840 27381 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-8deaa1f751034c15acd6ccaad37536c6 on tablet 8deaa1f751034c15acd6ccaad37536c6 with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:30.260680 30831 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:30.262948 27360 consensus_queue.cc:237] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:30.266153 30501 tablet_copy_client.cc:523] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:30.268455 30501 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:30.269763 30673 raft_consensus.cc:1050] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:30.269984 30969 ts_tablet_manager.cc:1875] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.270382 30673 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:30.270490 29117 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.271193 30876 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.272289 29250 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:30.272620 27093 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:30.272773 30908 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:30.273177 30907 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:30.275178 30805 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:30.275503 30805 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:30.275733 30805 ts_tablet_manager.cc:1367] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.472s	user 0.027s	sys 0.004s
I20240502 20:46:30.276214 30805 raft_consensus.cc:348] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:30.276347 30805 raft_consensus.cc:729] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:30.276165 30876 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.276440 30805 consensus_queue.cc:260] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:30.276739 30805 ts_tablet_manager.cc:1398] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:30.277446 27381 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-5586b9e60b544aed87c24f7b3ce0198c received from {username='slave'} at 127.19.66.136:44893
I20240502 20:46:30.277523 27381 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-5586b9e60b544aed87c24f7b3ce0198c on tablet 5586b9e60b544aed87c24f7b3ce0198c with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:30.278831 28976 raft_consensus.cc:1261] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:46:30.279013 28972 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.279047 27226 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:30.280297 30187 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.283514 29092 tablet_service.cc:1514] Processing DeleteTablet for tablet f00c4b7954564820875a25da5834416a with delete_type TABLET_DATA_TOMBSTONED (TS 1bc8a142cfc7487bacc573afb0a92b74 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54594
I20240502 20:46:30.284371 28980 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.286056 26810 catalog_manager.cc:5484] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:30.287283 28974 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.288254 26812 catalog_manager.cc:5484] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.289933 27358 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.290143 26960 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.290541 30513 raft_consensus.cc:348] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.290714 30513 raft_consensus.cc:729] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:30.290829 30513 consensus_queue.cc:260] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.291947 30513 ts_tablet_manager.cc:1398] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.040s	user 0.002s	sys 0.000s
I20240502 20:46:30.293630 30969 ts_tablet_manager.cc:1888] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:46:30.293745 30969 log.cc:1196] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 1f40a11333e44ea0a34b6bc7618af522: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/wals/62ebc6576a4e4a108de55a4e9d7ff6b8
I20240502 20:46:30.294621 30907 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:30.294682 30583 consensus_queue.cc:1035] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:46:30.295181 30194 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.295393 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-fdd2311ec0b645f5a79098a32c6deba2 received from {username='slave'} at 127.19.66.133:48853
I20240502 20:46:30.295477 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-fdd2311ec0b645f5a79098a32c6deba2 on tablet fdd2311ec0b645f5a79098a32c6deba2 with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:30.295825 30194 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.296895 27227 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.296959 30981 tablet_replica.cc:330] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: stopping tablet replica
I20240502 20:46:30.297487 29570 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.297783 28976 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.291322 28974 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.302353 30284 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.303368 27358 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:30.304047 28974 raft_consensus.cc:2932] T 078a1591d9d74e31a0407aeab76911b4 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.305246 27358 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.306340 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f00d12e425284c36ae8baed56c7cfaf6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.308655 30981 raft_consensus.cc:2229] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:30.309365 30981 pending_rounds.cc:62] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: Trying to abort 1 pending ops.
I20240502 20:46:30.309257 29117 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.315376 26957 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.318548 30580 raft_consensus.cc:1050] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:30.319192 30580 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:30.320432 27223 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.320948 29252 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:30.321434 30583 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:30.322511 27358 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:30.322811 26807 catalog_manager.cc:5484] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.323088 30583 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:30.324223 27089 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:30.325034 30027 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:30.326259 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet f00d12e425284c36ae8baed56c7cfaf6 with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:30.326522 30764 tablet_replica.cc:330] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:30.326695 30764 raft_consensus.cc:2229] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.326903 30764 raft_consensus.cc:2258] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.328382 30764 ts_tablet_manager.cc:1875] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.328671 30969 tablet_replica.cc:330] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: stopping tablet replica
I20240502 20:46:30.328850 30969 raft_consensus.cc:2229] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.329025 30969 raft_consensus.cc:2258] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.331259 30764 ts_tablet_manager.cc:1888] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:30.331370 30764 log.cc:1196] T f00d12e425284c36ae8baed56c7cfaf6 P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/f00d12e425284c36ae8baed56c7cfaf6
I20240502 20:46:30.332645 26794 catalog_manager.cc:4830] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 (table rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]) successfully deleted
I20240502 20:46:30.332875 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet f00d12e425284c36ae8baed56c7cfaf6 (table rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]) successfully deleted
I20240502 20:46:30.334692 30969 ts_tablet_manager.cc:1875] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.337489 28846 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.309567 30981 pending_rounds.cc:68] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023340708392943616 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f00c4b7954564820875a25da5834416a" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } } }
I20240502 20:46:30.338402 30981 raft_consensus.cc:2894] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240502 20:46:30.338927 27091 consensus_queue.cc:237] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:30.337847 30969 ts_tablet_manager.cc:1888] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 20:46:30.339762 30969 log.cc:1196] T bf189c19164442e5af0827f1fb7dc085 P 1f40a11333e44ea0a34b6bc7618af522: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/wals/bf189c19164442e5af0827f1fb7dc085
I20240502 20:46:30.340476 26794 catalog_manager.cc:4830] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): tablet bf189c19164442e5af0827f1fb7dc085 (table rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]) successfully deleted
I20240502 20:46:30.340840 30969 tablet_replica.cc:330] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: stopping tablet replica
I20240502 20:46:30.341367 29230 tablet_service.cc:1514] Processing DeleteTablet for tablet fdd2311ec0b645f5a79098a32c6deba2 with delete_type TABLET_DATA_TOMBSTONED (TS 6706d1c0946d42a9bae62d719e1e5c31 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:38022
I20240502 20:46:30.341622 26957 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.341692 30969 raft_consensus.cc:2229] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.341924 30969 pending_rounds.cc:62] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: Trying to abort 1 pending ops.
I20240502 20:46:30.342140 30981 raft_consensus.cc:2866] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 20:46:30.342267 30981 raft_consensus.cc:2258] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:30.342432 30058 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.342008 30969 pending_rounds.cc:68] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: Aborting op as it isn't in flight: id { term: 1 index: 11 } timestamp: 7023340708094414848 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "078a1591d9d74e31a0407aeab76911b4" old_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } } new_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } } }
I20240502 20:46:30.342703 30969 raft_consensus.cc:2894] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Aborting config change with OpId 1.11: Aborted: Op aborted
I20240502 20:46:30.342814 30969 raft_consensus.cc:2866] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 20:46:30.342885 30969 raft_consensus.cc:2258] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.343849 30969 ts_tablet_manager.cc:1875] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.347257 30969 ts_tablet_manager.cc:1888] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 20:46:30.347460 30969 log.cc:1196] T 078a1591d9d74e31a0407aeab76911b4 P 1f40a11333e44ea0a34b6bc7618af522: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/wals/078a1591d9d74e31a0407aeab76911b4
I20240502 20:46:30.348361 26794 catalog_manager.cc:4830] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): tablet 078a1591d9d74e31a0407aeab76911b4 (table rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]) successfully deleted
I20240502 20:46:30.349534 30981 ts_tablet_manager.cc:1875] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.357822 28980 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.359905 27226 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.361835 30058 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.368012 27358 consensus_queue.cc:237] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:30.370162 30579 raft_consensus.cc:1050] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:30.370358 30907 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.370684 30579 consensus_queue.cc:237] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.370925 28972 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.371654 30877 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.372725 28848 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.372758 27358 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.373237 30583 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.373699 30058 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.373773 27093 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.375067 29250 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.375823 30027 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.375967 30877 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.376247 29324 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.378325 30027 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.378625 26957 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.379231 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d44f271f166b4c288b76689a3e3764d3 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.380090 29117 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.380012 29250 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.380663 30712 consensus_queue.cc:1037] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.380730 27358 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.381558 29251 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.382041 29117 raft_consensus.cc:858] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
W20240502 20:46:30.382128 29117 raft_consensus.cc:1565] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:30.382557 29117 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:30.385243 26807 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.385950 28848 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.386471 26957 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.387229 30579 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:46:30.387846 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:30.387804 26801 catalog_manager.cc:5484] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:30.389204 26957 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.389801 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 843a26fc48844bfd82911b6a27833023 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.391547 30981 ts_tablet_manager.cc:1888] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:46:30.391777 30981 log.cc:1196] T f00c4b7954564820875a25da5834416a P 1bc8a142cfc7487bacc573afb0a92b74: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal/wals/f00c4b7954564820875a25da5834416a
I20240502 20:46:30.391971 27227 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.392000 28972 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.392752 26796 catalog_manager.cc:4830] TS 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): tablet f00c4b7954564820875a25da5834416a (table rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]) successfully deleted
I20240502 20:46:30.385632 26807 catalog_manager.cc:5484] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:30.396456 27227 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.396864 26806 catalog_manager.cc:5484] T 843a26fc48844bfd82911b6a27833023 P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.397363 30579 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.400171 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet 843a26fc48844bfd82911b6a27833023 with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:30.400483 30764 tablet_replica.cc:330] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:30.400339 29252 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.400667 30764 raft_consensus.cc:2229] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.400856 30764 raft_consensus.cc:2258] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.401193 28980 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.401790 30764 ts_tablet_manager.cc:1875] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.403218 27358 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.404446 30764 ts_tablet_manager.cc:1888] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:30.404548 30764 log.cc:1196] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/843a26fc48844bfd82911b6a27833023
I20240502 20:46:30.405292 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet 843a26fc48844bfd82911b6a27833023 (table rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]) successfully deleted
I20240502 20:46:30.409094 27089 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.411265 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet d44f271f166b4c288b76689a3e3764d3 with delete_type TABLET_DATA_TOMBSTONED (TS ebd8300729ec4226acc38b9cff2d8d3b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45924
I20240502 20:46:30.411602 30771 tablet_replica.cc:330] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b: stopping tablet replica
I20240502 20:46:30.411778 30771 raft_consensus.cc:2229] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.411998 30771 raft_consensus.cc:2258] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.412905 30771 ts_tablet_manager.cc:1875] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.414500 26806 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.416023 30771 ts_tablet_manager.cc:1888] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 20:46:30.416108 30771 log.cc:1196] T d44f271f166b4c288b76689a3e3764d3 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/wals/d44f271f166b4c288b76689a3e3764d3
I20240502 20:46:30.417240 26796 catalog_manager.cc:4830] TS ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): tablet d44f271f166b4c288b76689a3e3764d3 (table rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]) successfully deleted
I20240502 20:46:30.417882 27089 consensus_queue.cc:237] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:30.422696 30989 tablet_replica.cc:330] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: stopping tablet replica
I20240502 20:46:30.423208 30989 raft_consensus.cc:2229] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:30.423592 30989 pending_rounds.cc:62] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: Trying to abort 1 pending ops.
I20240502 20:46:30.423743 30989 pending_rounds.cc:68] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7023340708442791936 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "fdd2311ec0b645f5a79098a32c6deba2" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } } }
I20240502 20:46:30.424104 30989 raft_consensus.cc:2894] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240502 20:46:30.424255 30989 raft_consensus.cc:2866] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 20:46:30.424386 30989 raft_consensus.cc:2258] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:30.427521 30712 raft_consensus.cc:1050] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:30.428006 30712 consensus_queue.cc:237] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:30.428622 29117 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:30.431082 28848 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:30.431970 29324 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:30.433777 29115 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.434149 28972 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.434599 30147 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.434983 30147 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.436687 30513 ts_tablet_manager.cc:902] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:30.437287 30513 tablet_copy_client.cc:308] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:30.437397 28980 raft_consensus.cc:1261] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:30.438792 26957 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:30.440263 29252 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.440330 30147 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.440758 30673 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.442808 27089 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:30.443173 29324 consensus_queue.cc:1035] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:30.443226 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:48853)
I20240502 20:46:30.443336 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:48853: session id = 1f40a11333e44ea0a34b6bc7618af522-5704bdfe6c02437c8e9c8c3deaadb9e3
I20240502 20:46:30.444334 27112 tablet_copy_source_session.cc:205] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:30.444945 30673 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.444926 29115 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.446545 30513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:30.448819 29324 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.450326 30513 tablet_copy_client.cc:783] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:30.450811 30989 ts_tablet_manager.cc:1875] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.450584 30513 tablet_copy_client.cc:647] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:30.452216 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d44f271f166b4c288b76689a3e3764d3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.452749 26802 catalog_manager.cc:5484] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:30.452944 28974 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.452230 26801 catalog_manager.cc:5484] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 7 to 8, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.455600 28980 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.456614 30805 ts_tablet_manager.cc:902] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:30.457185 30805 tablet_copy_client.cc:308] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:38835
I20240502 20:46:30.457755 26980 tablet_copy_service.cc:140] P 297de823852d4a8da44791ab03131cd3: Received BeginTabletCopySession request for tablet f2dfc0bbbbca4cc28a97480007343424 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:35489)
I20240502 20:46:30.457628 28848 raft_consensus.cc:2932] T d44f271f166b4c288b76689a3e3764d3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.457897 26980 tablet_copy_service.cc:161] P 297de823852d4a8da44791ab03131cd3: Beginning new tablet copy session on tablet f2dfc0bbbbca4cc28a97480007343424 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:35489: session id = 6706d1c0946d42a9bae62d719e1e5c31-f2dfc0bbbbca4cc28a97480007343424
I20240502 20:46:30.459009 26980 tablet_copy_source_session.cc:205] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:30.460027 30805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2dfc0bbbbca4cc28a97480007343424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:30.461212 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet d44f271f166b4c288b76689a3e3764d3 with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:30.461503 30764 tablet_replica.cc:330] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:30.461702 30764 raft_consensus.cc:2229] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.461977 30764 raft_consensus.cc:2258] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.463084 30805 tablet_copy_client.cc:783] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:30.463093 30764 ts_tablet_manager.cc:1875] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.463244 30805 tablet_copy_client.cc:647] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:30.467068 30764 ts_tablet_manager.cc:1888] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:46:30.467325 30764 log.cc:1196] T d44f271f166b4c288b76689a3e3764d3 P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/d44f271f166b4c288b76689a3e3764d3
I20240502 20:46:30.468248 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet d44f271f166b4c288b76689a3e3764d3 (table rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]) successfully deleted
I20240502 20:46:30.478096 30989 ts_tablet_manager.cc:1888] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 20:46:30.481302 30989 log.cc:1196] T fdd2311ec0b645f5a79098a32c6deba2 P 6706d1c0946d42a9bae62d719e1e5c31: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-7/wal/wals/fdd2311ec0b645f5a79098a32c6deba2
I20240502 20:46:30.482277 26795 catalog_manager.cc:4830] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): tablet fdd2311ec0b645f5a79098a32c6deba2 (table rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]) successfully deleted
I20240502 20:46:30.496371 27227 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:30.498236 30877 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:30.498965 30877 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:30.499089 30489 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:30.499495 30489 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:30.500906 27227 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.504272 27226 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:30.507234 30850 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:30.508591 30489 ts_tablet_manager.cc:1367] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.542s	user 0.039s	sys 0.000s
I20240502 20:46:30.509780 30489 raft_consensus.cc:348] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:30.511690 30489 raft_consensus.cc:729] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:30.511947 30489 consensus_queue.cc:260] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:30.508270 30465 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:30.513897 30465 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:30.514252 30465 ts_tablet_manager.cc:1367] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.547s	user 0.034s	sys 0.000s
I20240502 20:46:30.514475 30673 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.515911 30465 raft_consensus.cc:348] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:30.516520 30465 raft_consensus.cc:729] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:30.516577 26794 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af693f5816e046f592fbd9b2e7ef0ad9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.516718 30465 consensus_queue.cc:260] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:30.516430 30489 ts_tablet_manager.cc:1398] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.000s
I20240502 20:46:30.517160 29252 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.517942 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-367543df62e048268908eb0d84c14876 received from {username='slave'} at 127.19.66.134:40783
I20240502 20:46:30.518059 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-367543df62e048268908eb0d84c14876 on tablet 367543df62e048268908eb0d84c14876 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:30.519176 30465 ts_tablet_manager.cc:1398] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.000s
I20240502 20:46:30.521294 30850 tablet_bootstrap.cc:492] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:30.522607 26980 tablet_copy_service.cc:342] P 297de823852d4a8da44791ab03131cd3: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-a2b61909194d488682afd91c0f796405 received from {username='slave'} at 127.19.66.134:52999
I20240502 20:46:30.522738 26980 tablet_copy_service.cc:433] P 297de823852d4a8da44791ab03131cd3: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-a2b61909194d488682afd91c0f796405 on tablet a2b61909194d488682afd91c0f796405 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:30.522748 27089 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.523751 26802 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:30.527684 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet af693f5816e046f592fbd9b2e7ef0ad9 with delete_type TABLET_DATA_TOMBSTONED (TS c107c43e538d48f58625c4320f0aa4b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43554
I20240502 20:46:30.530372 30850 ts_tablet_manager.cc:1367] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.639s	user 0.035s	sys 0.004s
I20240502 20:46:30.531095 30850 raft_consensus.cc:348] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:30.531332 30850 raft_consensus.cc:729] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:30.531498 30850 consensus_queue.cc:260] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:30.534610 30437 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:30.535038 30437 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:30.535297 30437 ts_tablet_manager.cc:1367] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.449s	user 0.035s	sys 0.000s
I20240502 20:46:30.536103 30850 ts_tablet_manager.cc:1398] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.001s
I20240502 20:46:30.536767 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-367543df62e048268908eb0d84c14876 received from {username='slave'} at 127.19.66.135:52059
I20240502 20:46:30.536875 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-367543df62e048268908eb0d84c14876 on tablet 367543df62e048268908eb0d84c14876 with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:30.538652 30437 raft_consensus.cc:348] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:30.538930 30437 raft_consensus.cc:729] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:30.539155 31039 tablet_replica.cc:330] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: stopping tablet replica
I20240502 20:46:30.539125 30437 consensus_queue.cc:260] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:30.539369 31039 raft_consensus.cc:2229] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.539594 31039 raft_consensus.cc:2258] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.539851 30437 ts_tablet_manager.cc:1398] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240502 20:46:30.540603 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-84484d65e2bb4e23ac621a39434ce382 received from {username='slave'} at 127.19.66.135:49537
I20240502 20:46:30.540707 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-84484d65e2bb4e23ac621a39434ce382 on tablet 84484d65e2bb4e23ac621a39434ce382 with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:30.542132 31039 ts_tablet_manager.cc:1875] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.556453 31039 ts_tablet_manager.cc:1888] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:30.556691 31039 log.cc:1196] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/af693f5816e046f592fbd9b2e7ef0ad9
I20240502 20:46:30.557593 26797 catalog_manager.cc:4830] TS c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): tablet af693f5816e046f592fbd9b2e7ef0ad9 (table rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]) successfully deleted
I20240502 20:46:30.565595 30513 tablet_copy_client.cc:523] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:30.570677 30513 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:30.616405 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.626416 30805 tablet_copy_client.cc:523] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:30.628476 30805 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:30.650836 30501 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:30.651347 30501 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:30.652040 30501 ts_tablet_manager.cc:1367] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.384s	user 0.026s	sys 0.005s
I20240502 20:46:30.653884 30501 raft_consensus.cc:348] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.654333 30501 raft_consensus.cc:729] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:30.654565 30501 consensus_queue.cc:260] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.655648 30501 ts_tablet_manager.cc:1398] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:46:30.656296 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-b5f81e46ab1f48fc8a88495b42f83640 received from {username='slave'} at 127.19.66.133:35399
I20240502 20:46:30.656394 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-b5f81e46ab1f48fc8a88495b42f83640 on tablet b5f81e46ab1f48fc8a88495b42f83640 with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:30.661254 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.681612 28825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.683780 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.693739 27089 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:30.695786 27227 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:30.696609 30194 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:30.696679 28972 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:30.697506 26957 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:30.697995 30194 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:30.699831 30194 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:30.701978 29252 raft_consensus.cc:1203] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:30.703891 30058 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.708447 27227 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.709583 28972 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.713307 26801 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.713315 26957 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.716919 28825 tablet_service.cc:1514] Processing DeleteTablet for tablet fdd2311ec0b645f5a79098a32c6deba2 with delete_type TABLET_DATA_TOMBSTONED (TS 1f40a11333e44ea0a34b6bc7618af522 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47924
I20240502 20:46:30.717351 30969 tablet_replica.cc:330] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: stopping tablet replica
I20240502 20:46:30.717586 30969 raft_consensus.cc:2229] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:30.717720 30969 raft_consensus.cc:2258] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:30.718506 30969 ts_tablet_manager.cc:1875] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.723690 30969 ts_tablet_manager.cc:1888] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:46:30.724066 30969 log.cc:1196] T fdd2311ec0b645f5a79098a32c6deba2 P 1f40a11333e44ea0a34b6bc7618af522: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-4/wal/wals/fdd2311ec0b645f5a79098a32c6deba2
I20240502 20:46:30.725389 26794 catalog_manager.cc:4830] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): tablet fdd2311ec0b645f5a79098a32c6deba2 (table rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]) successfully deleted
I20240502 20:46:30.730209 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fdd2311ec0b645f5a79098a32c6deba2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.736991 27089 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:30.740303 28972 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:30.740764 26957 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:30.741163 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:30.741307 30194 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:30.741721 30194 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:30.743863 30058 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.747661 26810 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 9 to 10, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:30.747776 26957 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.751611 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.753902 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet fdd2311ec0b645f5a79098a32c6deba2 with delete_type TABLET_DATA_TOMBSTONED (TS c107c43e538d48f58625c4320f0aa4b8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43554
I20240502 20:46:30.754725 31039 tablet_replica.cc:330] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: stopping tablet replica
I20240502 20:46:30.754904 31039 raft_consensus.cc:2229] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:30.755185 31039 raft_consensus.cc:2258] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:30.755390 28972 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:30.756100 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fdd2311ec0b645f5a79098a32c6deba2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.759639 31039 ts_tablet_manager.cc:1875] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:30.762560 31039 ts_tablet_manager.cc:1888] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 20:46:30.762666 31039 log.cc:1196] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/fdd2311ec0b645f5a79098a32c6deba2
I20240502 20:46:30.776782 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:30.786238 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:30.786815 28825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.790617 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:30.792783 30805 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:30.805241 30805 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:30.805797 30805 ts_tablet_manager.cc:1367] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.177s	user 0.031s	sys 0.005s
I20240502 20:46:30.809032 30805 raft_consensus.cc:348] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:30.809378 30805 raft_consensus.cc:729] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:30.809634 30805 consensus_queue.cc:260] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:30.810367 26797 catalog_manager.cc:4830] TS c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): tablet fdd2311ec0b645f5a79098a32c6deba2 (table rebalance_test_table_5 [id=01e571f10ffe4d4c88601420e733e5c7]) successfully deleted
I20240502 20:46:30.813046 30805 ts_tablet_manager.cc:1398] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.001s
I20240502 20:46:30.814306 26980 tablet_copy_service.cc:342] P 297de823852d4a8da44791ab03131cd3: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-f2dfc0bbbbca4cc28a97480007343424 received from {username='slave'} at 127.19.66.136:35489
I20240502 20:46:30.814421 26980 tablet_copy_service.cc:433] P 297de823852d4a8da44791ab03131cd3: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-f2dfc0bbbbca4cc28a97480007343424 on tablet f2dfc0bbbbca4cc28a97480007343424 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:30.830756 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:30.857803 28972 raft_consensus.cc:1203] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 20:46:30.862608 29252 raft_consensus.cc:1203] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:30.866283 29115 raft_consensus.cc:1203] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 20:46:30.872345 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.875284 28825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.884958 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.885720 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:30.897279 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:30.897719 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.915529 29115 raft_consensus.cc:1203] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:30.916605 28848 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.917879 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:30.924537 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:30.942432 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.948578 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:30.950672 27227 consensus_queue.cc:237] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:30.953025 29252 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.953675 30582 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.955073 26957 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:30.956117 30582 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:30.958207 30582 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.963254 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5f81e46ab1f48fc8a88495b42f83640 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:30.963677 29252 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.964465 26807 catalog_manager.cc:5484] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 6 to 7, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:30.968761 26957 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:30.970350 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:30.975767 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:30.981845 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet b5f81e46ab1f48fc8a88495b42f83640 with delete_type TABLET_DATA_TOMBSTONED (TS ebd8300729ec4226acc38b9cff2d8d3b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45924
I20240502 20:46:30.984052 28972 raft_consensus.cc:1203] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 20:46:30.995669 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:31.004406 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:31.019076 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:31.020635 31165 tablet_replica.cc:330] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b: stopping tablet replica
I20240502 20:46:31.021585 31165 raft_consensus.cc:2229] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.023116 31165 raft_consensus.cc:2258] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.034750 31165 ts_tablet_manager.cc:1875] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.042667 31165 ts_tablet_manager.cc:1888] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 20:46:31.042896 31165 log.cc:1196] T b5f81e46ab1f48fc8a88495b42f83640 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/wals/b5f81e46ab1f48fc8a88495b42f83640
I20240502 20:46:31.043565 28846 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.044195 30582 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.045045 26796 catalog_manager.cc:4830] TS ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): tablet b5f81e46ab1f48fc8a88495b42f83640 (table rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]) successfully deleted
I20240502 20:46:31.045269 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:31.045951 28846 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.057783 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:31.068473 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:31.099608 30248 raft_consensus.cc:1050] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 to VOTER
I20240502 20:46:31.100181 30248 consensus_queue.cc:237] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } }
I20240502 20:46:31.101784 29252 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.101982 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:31.103523 30228 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.103684 26957 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.103891 28846 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.104333 30228 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.104667 30228 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.108259 30579 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.108979 29252 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.111621 26957 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.112104 26806 catalog_manager.cc:5484] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.129580 27227 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:31.132180 27089 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.132346 27358 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.133416 30579 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.133898 30579 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.136101 30228 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.139431 27358 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.142892 27089 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.143323 26801 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.148844 29252 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.152612 28825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:31.152724 28846 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.160714 30579 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.162515 29252 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.168520 27226 consensus_queue.cc:237] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } }
I20240502 20:46:31.168534 27227 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.173414 27358 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.174185 30582 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.178223 28848 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.178810 31180 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.179764 29251 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.179944 27089 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:46:31.180936 27158 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a547561343304137a044ce11517f6007. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.182444 30228 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.183938 31181 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.185439 29252 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.185628 27358 consensus_queue.cc:237] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:31.187656 29117 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.187659 31180 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.188298 30582 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.188755 30582 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.188890 30284 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.191568 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5f81e46ab1f48fc8a88495b42f83640 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:31.192415 27093 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.192826 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a547561343304137a044ce11517f6007 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:46:31.196058 27089 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.196439 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:31.196235 29252 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.196564 27358 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.198693 26957 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.198864 26810 catalog_manager.cc:5484] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:31.199616 29251 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.202078 30284 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.202626 30877 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.203018 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet b5f81e46ab1f48fc8a88495b42f83640 with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:31.203476 31185 tablet_replica.cc:330] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:31.203647 31185 raft_consensus.cc:2229] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.203893 31185 raft_consensus.cc:2258] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.203552 28848 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.206387 26807 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.206653 31185 ts_tablet_manager.cc:1875] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.210144 31185 ts_tablet_manager.cc:1888] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:46:31.210323 31185 log.cc:1196] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/b5f81e46ab1f48fc8a88495b42f83640
I20240502 20:46:31.211007 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet b5f81e46ab1f48fc8a88495b42f83640 (table rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]) successfully deleted
I20240502 20:46:31.213500 27227 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.215584 27089 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.216245 31181 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 20:46:31.216526 27158 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a547561343304137a044ce11517f6007. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.217192 29251 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.218078 31181 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.220738 30513 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:31.221238 30513 tablet_bootstrap.cc:492] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:31.221518 30513 ts_tablet_manager.cc:1367] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.651s	user 0.038s	sys 0.000s
I20240502 20:46:31.221356 30248 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.222128 29251 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.224345 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a547561343304137a044ce11517f6007 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:31.222154 30513 raft_consensus.cc:348] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.225170 30513 raft_consensus.cc:729] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:31.225356 30513 consensus_queue.cc:260] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.226922 30513 ts_tablet_manager.cc:1398] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.000s
I20240502 20:46:31.227763 27089 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.228149 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-5704bdfe6c02437c8e9c8c3deaadb9e3 received from {username='slave'} at 127.19.66.133:48853
I20240502 20:46:31.228236 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-5704bdfe6c02437c8e9c8c3deaadb9e3 on tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:31.235314 30877 raft_consensus.cc:1050] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:31.235896 30877 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:31.238905 27093 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.240038 26801 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 7 to 8, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.243088 30284 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.245420 30284 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.248327 29117 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.249719 27091 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.255429 26805 catalog_manager.cc:5484] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.256518 27227 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.260725 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet a547561343304137a044ce11517f6007 with delete_type TABLET_DATA_TOMBSTONED (TS ebd8300729ec4226acc38b9cff2d8d3b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45924
I20240502 20:46:31.261149 31165 tablet_replica.cc:330] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: stopping tablet replica
I20240502 20:46:31.261345 31165 raft_consensus.cc:2229] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.261601 31165 raft_consensus.cc:2258] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.264004 30831 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.264153 31165 ts_tablet_manager.cc:1875] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.265313 27358 consensus_queue.cc:237] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.265368 26957 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.267654 29117 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.268625 30907 raft_consensus.cc:1050] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:31.268802 31190 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.269114 30907 consensus_queue.cc:237] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:31.269608 30058 raft_consensus.cc:1050] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 to VOTER
I20240502 20:46:31.270180 30058 consensus_queue.cc:237] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:31.271243 29252 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.271442 29251 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.271828 31199 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.271914 30877 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.272220 29117 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.272819 30064 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.273046 31165 ts_tablet_manager.cc:1888] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 20:46:31.273162 31165 log.cc:1196] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/wals/a547561343304137a044ce11517f6007
I20240502 20:46:31.274087 28848 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.274295 26957 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.274384 27226 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.274637 28972 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.275015 31199 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:31.275347 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a8ce1df86914b2caf2d083d97eda4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.275717 26796 catalog_manager.cc:4830] TS ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): tablet a547561343304137a044ce11517f6007 (table rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]) successfully deleted
I20240502 20:46:31.276418 30064 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.276746 30064 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.277791 30831 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.278422 30831 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.280066 27093 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.281682 27358 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.282709 29251 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.283064 30877 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.284081 26957 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.285329 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6a8ce1df86914b2caf2d083d97eda4e6 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:46:31.285655 30194 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.286047 27091 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.287135 31190 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.287729 29115 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.290524 27093 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.291229 26809 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.293874 26812 catalog_manager.cc:5484] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.294669 27227 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.294016 27089 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.295796 30064 raft_consensus.cc:1013] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: attempt to promote peer 1bc8a142cfc7487bacc573afb0a92b74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 20:46:31.299805 30908 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.300406 30413 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.301875 27358 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:31.303843 29252 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.303601 29251 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.304749 31190 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.306674 27226 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.307008 26810 catalog_manager.cc:5484] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.307849 27089 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.308910 31190 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.312522 30877 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.314745 27089 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.314904 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5586b9e60b544aed87c24f7b3ce0198c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:31.315354 26957 consensus_queue.cc:237] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:31.316169 27093 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.317939 29251 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.318513 30907 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.319916 27226 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.320472 30907 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.321542 29252 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.322757 26812 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.325067 30907 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.326820 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet 5586b9e60b544aed87c24f7b3ce0198c with delete_type TABLET_DATA_TOMBSTONED (TS c107c43e538d48f58625c4320f0aa4b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43554
I20240502 20:46:31.329293 29252 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.332729 27226 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:31.333771 27227 consensus_queue.cc:237] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.335023 26794 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2dfc0bbbbca4cc28a97480007343424 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:31.336388 26810 catalog_manager.cc:5484] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:31.337872 31218 tablet_replica.cc:330] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: stopping tablet replica
I20240502 20:46:31.338151 31218 raft_consensus.cc:2229] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.338476 31218 raft_consensus.cc:2258] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.340474 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet f2dfc0bbbbca4cc28a97480007343424 with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47758
I20240502 20:46:31.345937 31218 ts_tablet_manager.cc:1875] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.345980 26957 raft_consensus.cc:1261] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:31.345793 31201 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
W20240502 20:46:31.346628 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62c71a33fbcc4cfdbf0065a3fc9dfeba. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.346834 30579 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:31.346930 28980 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:31.349752 31218 ts_tablet_manager.cc:1888] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:31.349982 31218 log.cc:1196] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/5586b9e60b544aed87c24f7b3ce0198c
I20240502 20:46:31.350833 26797 catalog_manager.cc:4830] TS c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): tablet 5586b9e60b544aed87c24f7b3ce0198c (table rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]) successfully deleted
I20240502 20:46:31.350836 29117 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:31.353673 27089 raft_consensus.cc:1261] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:31.353811 27358 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:31.354319 31181 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:31.354461 26801 catalog_manager.cc:5484] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 8 to 9, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240502 20:46:31.359289 28848 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:31.362706 27093 consensus_queue.cc:237] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:31.365164 28848 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:31.365518 28980 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:31.365963 30064 raft_consensus.cc:1013] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: attempt to promote peer 1bc8a142cfc7487bacc573afb0a92b74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 20:46:31.367609 27358 raft_consensus.cc:1261] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:31.368119 30064 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:31.371133 31201 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:31.371555 30194 consensus_queue.cc:1035] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:31.371942 31220 tablet_replica.cc:330] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:46:31.372885 31220 raft_consensus.cc:2229] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.377463 31181 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.390120 26801 catalog_manager.cc:5484] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.391803 26957 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.402449 30579 raft_consensus.cc:1050] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:31.402992 30579 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:31.405618 31220 raft_consensus.cc:2258] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.406121 30058 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:31.406572 27093 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.407048 29117 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.407532 30582 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.407699 31220 ts_tablet_manager.cc:1875] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.408465 26957 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.408885 27358 consensus_queue.cc:237] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:31.409720 28848 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:31.410306 30582 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:31.411198 27296 consensus_peers.cc:485] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14a7bead40fb4eb0bec6b00804b52595. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.412161 31220 ts_tablet_manager.cc:1888] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:31.412254 31220 log.cc:1196] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/f2dfc0bbbbca4cc28a97480007343424
I20240502 20:46:31.412570 28980 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:31.412792 28974 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.412842 26960 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.414502 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 367543df62e048268908eb0d84c14876 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:31.414748 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet f2dfc0bbbbca4cc28a97480007343424 (table rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]) successfully deleted
I20240502 20:46:31.415221 31211 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.415772 31210 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.416173 26812 catalog_manager.cc:5484] T 367543df62e048268908eb0d84c14876 P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:31.420312 27093 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.420727 29092 tablet_service.cc:1514] Processing DeleteTablet for tablet 367543df62e048268908eb0d84c14876 with delete_type TABLET_DATA_TOMBSTONED (TS 1bc8a142cfc7487bacc573afb0a92b74 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54594
I20240502 20:46:31.421041 31181 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.422320 27358 raft_consensus.cc:2932] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:31.422880 28848 raft_consensus.cc:1203] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 20:46:31.424130 30908 raft_consensus.cc:1050] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3: attempting to promote NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 to VOTER
I20240502 20:46:31.424736 30908 consensus_queue.cc:237] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:31.426770 28980 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.426833 27226 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.427413 30413 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.427875 30413 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.430507 27361 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.432940 30907 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.437331 31181 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.439092 31211 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.442044 29117 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.442487 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:31.442413 26812 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:31.445469 26960 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.445839 28980 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.456722 27226 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:31.457176 26807 catalog_manager.cc:5484] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 6 to 7, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.459103 26960 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.459051 26957 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.460788 30228 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.467415 30228 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.469854 29117 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.470683 26807 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.470893 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:31.471089 30248 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.471208 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84484d65e2bb4e23ac621a39434ce382 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:31.472441 29117 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.472637 26960 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.475960 27226 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.478240 27089 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 20:46:31.478276 27158 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a547561343304137a044ce11517f6007. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.478894 30248 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.479594 29252 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.481601 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:31.483709 30228 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.485623 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet 84484d65e2bb4e23ac621a39434ce382 with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47758
I20240502 20:46:31.485926 31220 tablet_replica.cc:330] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:46:31.486104 31220 raft_consensus.cc:2229] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.486333 31220 pending_rounds.cc:62] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Trying to abort 1 pending ops.
I20240502 20:46:31.486372 31220 pending_rounds.cc:68] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023340713585033216 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "84484d65e2bb4e23ac621a39434ce382" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } } }
I20240502 20:46:31.486544 31220 raft_consensus.cc:2894] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 20:46:31.486613 31220 raft_consensus.cc:2866] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 20:46:31.486656 31220 raft_consensus.cc:2258] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.486860 30228 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.489398 27089 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.491801 31264 tablet_replica.cc:330] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: stopping tablet replica
I20240502 20:46:31.493140 30907 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:31.494002 31264 raft_consensus.cc:2229] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 20:46:31.494187 31264 raft_consensus.cc:2258] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 20:46:31.495940 27226 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
W20240502 20:46:31.501325 31272 consensus_queue.cc:1159] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d1f3f8e444364d6faa42d651b5dbb997" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240502 20:46:31.501511 28825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:31.495710 31264 ts_tablet_manager.cc:1875] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.501420 26810 catalog_manager.cc:5484] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:31.503355 28980 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:31.503777 26801 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.506954 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:31.510308 27358 consensus_queue.cc:237] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:31.510468 29252 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.513098 27227 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
W20240502 20:46:31.515144 27158 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a547561343304137a044ce11517f6007. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.515174 27358 consensus_queue.cc:237] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:31.517138 31220 ts_tablet_manager.cc:1875] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.518471 27358 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
W20240502 20:46:31.518910 27158 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a547561343304137a044ce11517f6007. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:31.519282 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f00d12e425284c36ae8baed56c7cfaf6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.520251 27093 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:31.520568 29251 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:31.521024 31220 ts_tablet_manager.cc:1888] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:31.521235 31220 log.cc:1196] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/84484d65e2bb4e23ac621a39434ce382
I20240502 20:46:31.522214 30831 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:31.522270 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet 84484d65e2bb4e23ac621a39434ce382 (table rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]) successfully deleted
I20240502 20:46:31.523579 29117 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.524273 31190 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.526471 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:31.526891 31264 ts_tablet_manager.cc:1888] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 20:46:31.527066 27227 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.528424 26957 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:31.529006 30831 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:31.529595 31264 log.cc:1196] T 367543df62e048268908eb0d84c14876 P 1bc8a142cfc7487bacc573afb0a92b74: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-6/wal/wals/367543df62e048268908eb0d84c14876
I20240502 20:46:31.529976 30248 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:31.530548 26796 catalog_manager.cc:4830] TS 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): tablet 367543df62e048268908eb0d84c14876 (table rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]) successfully deleted
I20240502 20:46:31.530548 30831 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.531771 27091 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:31.532425 30248 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:46:31.533275 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40df15a82a624913b3b712687d999134. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.535266 26959 consensus_queue.cc:237] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:31.536383 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:31.539454 28972 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.539448 30228 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.541877 27089 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.544957 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a547561343304137a044ce11517f6007 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:46:31.545055 30908 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.545804 29252 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.546370 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:31.547984 31189 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.548573 26960 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.552256 26810 catalog_manager.cc:5484] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.554332 26809 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 9 to 10, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.560845 27358 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:46:31.561466 30908 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:46:31.567584 26807 catalog_manager.cc:5031] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a547561343304137a044ce11517f6007 with cas_config_opid_index 9: aborting the task: latest config opid_index 10; task opid_index 9
I20240502 20:46:31.570595 30907 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:31.573882 26794 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2b61909194d488682afd91c0f796405 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:31.574477 27358 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:31.576639 26810 catalog_manager.cc:5484] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:31.581220 27091 consensus_queue.cc:237] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:31.582696 28972 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } }
I20240502 20:46:31.584939 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet a2b61909194d488682afd91c0f796405 with delete_type TABLET_DATA_TOMBSTONED (TS c107c43e538d48f58625c4320f0aa4b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43554
I20240502 20:46:31.585301 31218 tablet_replica.cc:330] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: stopping tablet replica
I20240502 20:46:31.585494 31218 raft_consensus.cc:2229] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.585726 31218 raft_consensus.cc:2258] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.585924 30831 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
W20240502 20:46:31.588018 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff9fef57f6eb490da536170a02b96212. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.588239 31218 ts_tablet_manager.cc:1875] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.588369 28972 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.588829 30058 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.589592 29115 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.591034 31218 ts_tablet_manager.cc:1888] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 20:46:31.591116 31218 log.cc:1196] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/a2b61909194d488682afd91c0f796405
I20240502 20:46:31.591969 26797 catalog_manager.cc:4830] TS c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): tablet a2b61909194d488682afd91c0f796405 (table rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]) successfully deleted
I20240502 20:46:31.592202 27358 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.593037 30058 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.594616 26801 catalog_manager.cc:5484] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.596870 27093 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.602270 30058 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.605327 28980 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.607527 26809 catalog_manager.cc:5484] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 6 to 7, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.617512 27227 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.627453 27358 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.635562 27358 consensus_queue.cc:237] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.636569 27361 consensus_queue.cc:237] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.636909 27093 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
W20240502 20:46:31.638854 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f00d12e425284c36ae8baed56c7cfaf6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.639005 29252 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.639043 27359 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:46:31.639321 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1b9ff3c921d474fa5ad106d1c17ab00. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.639858 29115 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.639885 30064 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:46:31.640223 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40df15a82a624913b3b712687d999134. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:31.640951 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f00d12e425284c36ae8baed56c7cfaf6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.641546 26960 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.641630 27227 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.642021 31190 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.642277 30831 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.643150 30831 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.643447 27093 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.644309 31210 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.646091 30064 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:46:31.647239 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40df15a82a624913b3b712687d999134. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.648322 26960 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.654497 30064 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.656163 29252 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.658521 26810 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.663597 31211 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.664904 30831 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.666030 29115 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.666069 27227 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.666366 27358 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.671803 26807 catalog_manager.cc:5484] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.673286 27358 consensus_queue.cc:237] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:31.674990 28972 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.675702 31211 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.676168 26809 catalog_manager.cc:5484] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.678128 29115 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.679375 30284 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.681807 31210 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.682525 27227 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.682591 29115 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.685110 27093 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.685657 28972 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.686553 30877 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.688457 27226 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.689617 26807 catalog_manager.cc:5484] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:31.691622 27227 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
W20240502 20:46:31.694202 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a906569ab3b14520b4876bc8d5450f7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.694442 28848 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.695000 27358 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:31.695411 30582 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.695665 30228 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:31.701875 31180 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:31.702538 28848 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:31.706372 27359 consensus_queue.cc:237] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.707731 27361 consensus_queue.cc:237] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:31.708273 26960 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.710186 29117 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:31.710640 28972 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:31.710789 26812 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.711035 30831 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.711436 29115 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 20:46:31.711867 27296 consensus_peers.cc:485] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8deaa1f751034c15acd6ccaad37536c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.712075 31211 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:31.712384 31211 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.712745 27226 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:31.714488 31190 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:31.714859 31190 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:31.718185 27358 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:31.720153 30877 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.722482 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a8ce1df86914b2caf2d083d97eda4e6 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 20:46:31.723102 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a8ce1df86914b2caf2d083d97eda4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.724112 29117 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.725018 27226 consensus_queue.cc:237] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:31.725661 26960 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.727025 27358 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:31.728099 30228 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:31.729696 31211 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.730458 26957 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 20:46:31.731160 26812 catalog_manager.cc:5484] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 7 to 8, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.732232 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8deaa1f751034c15acd6ccaad37536c6 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 20:46:31.732285 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd1b1316da2d47938164ac5b75f11bae. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.732543 30228 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 20:46:31.732839 27226 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.733886 28972 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.734884 27227 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
W20240502 20:46:31.737231 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf189c19164442e5af0827f1fb7dc085. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.737290 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:31.737905 29252 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:31.738646 30582 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:31.738690 26801 catalog_manager.cc:5484] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 reported cstate change: config changed from index 9 to 10, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.739384 30228 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:31.741255 30228 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.742842 29115 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.744719 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a8ce1df86914b2caf2d083d97eda4e6 with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47758
I20240502 20:46:31.745031 31220 tablet_replica.cc:330] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:46:31.745261 31220 raft_consensus.cc:2229] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.745513 31220 raft_consensus.cc:2258] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.746553 31220 ts_tablet_manager.cc:1875] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.748860 27358 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.749822 31220 ts_tablet_manager.cc:1888] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 20:46:31.750013 31220 log.cc:1196] T 6a8ce1df86914b2caf2d083d97eda4e6 P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/6a8ce1df86914b2caf2d083d97eda4e6
I20240502 20:46:31.750268 27359 consensus_queue.cc:237] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:31.750603 26960 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:46:31.751350 26809 catalog_manager.cc:5484] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.752599 28972 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:46:31.753019 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet 6a8ce1df86914b2caf2d083d97eda4e6 (table rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]) successfully deleted
I20240502 20:46:31.753341 31210 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:46:31.754992 27296 consensus_peers.cc:485] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8deaa1f751034c15acd6ccaad37536c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.755540 29115 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:46:31.756152 31211 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:46:31.756362 31190 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.759235 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8deaa1f751034c15acd6ccaad37536c6 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:31.759724 28972 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.759922 26807 catalog_manager.cc:5484] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 10 to 11, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.761890 29115 raft_consensus.cc:2932] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:31.767100 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet 8deaa1f751034c15acd6ccaad37536c6 with delete_type TABLET_DATA_TOMBSTONED (TS c107c43e538d48f58625c4320f0aa4b8 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:43554
I20240502 20:46:31.767434 31218 tablet_replica.cc:330] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8: stopping tablet replica
I20240502 20:46:31.767676 31218 raft_consensus.cc:2229] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.767993 31218 raft_consensus.cc:2258] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.769878 30228 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:31.772503 31218 ts_tablet_manager.cc:1875] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.773535 27361 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:31.775875 31218 ts_tablet_manager.cc:1888] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 20:46:31.776031 31218 log.cc:1196] T 8deaa1f751034c15acd6ccaad37536c6 P c107c43e538d48f58625c4320f0aa4b8: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/8deaa1f751034c15acd6ccaad37536c6
I20240502 20:46:31.777168 26797 catalog_manager.cc:4830] TS c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): tablet 8deaa1f751034c15acd6ccaad37536c6 (table rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]) successfully deleted
I20240502 20:46:31.777838 26809 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.783455 29252 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:31.792433 27226 consensus_queue.cc:237] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.795293 26960 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:46:31.795651 27361 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 20:46:31.796166 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd1b1316da2d47938164ac5b75f11bae. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.796265 31180 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:46:31.796387 30582 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 20:46:31.797055 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd1b1316da2d47938164ac5b75f11bae. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.799024 30582 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.799801 31223 raft_consensus.cc:1050] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 to VOTER
I20240502 20:46:31.800369 31223 consensus_queue.cc:237] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } }
I20240502 20:46:31.802413 29115 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.802978 26960 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.803846 26809 catalog_manager.cc:5484] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.805158 27359 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.805521 31202 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.806756 30194 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.810853 28848 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.812041 31202 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.814107 27361 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:31.814767 27093 consensus_queue.cc:237] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
W20240502 20:46:31.817373 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff9fef57f6eb490da536170a02b96212. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.818382 28972 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 20:46:31.820489 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:31.820784 31202 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.825198 31201 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:31.825497 27361 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:31.825973 31202 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:31.826200 28972 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:31.829540 26812 catalog_manager.cc:5484] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 7 to 8, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:31.831995 27361 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:31.838410 30058 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.839494 29115 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.841979 28848 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.842361 27359 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.842828 26812 catalog_manager.cc:5484] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 7 to 8, 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:31.848358 27093 consensus_queue.cc:237] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } }
I20240502 20:46:31.850344 28848 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.850940 31202 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.853394 30058 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.853711 29117 raft_consensus.cc:1261] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:31.853915 28848 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.854179 31201 consensus_queue.cc:1035] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:31.855394 29117 raft_consensus.cc:2932] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.857210 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:31.857019 26807 catalog_manager.cc:5484] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P 1f40a11333e44ea0a34b6bc7618af522 reported cstate change: config changed from index 8 to 9, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:46:31.861431 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 with delete_type TABLET_DATA_TOMBSTONED (TS ebd8300729ec4226acc38b9cff2d8d3b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45924
I20240502 20:46:31.865671 31441 tablet_replica.cc:330] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b: stopping tablet replica
I20240502 20:46:31.865983 31441 raft_consensus.cc:2229] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:31.866345 31441 raft_consensus.cc:2258] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:31.867221 31441 ts_tablet_manager.cc:1875] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:31.870230 31441 ts_tablet_manager.cc:1888] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:46:31.871250 31441 log.cc:1196] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/wals/5704bdfe6c02437c8e9c8c3deaadb9e3
I20240502 20:46:31.872107 26796 catalog_manager.cc:4830] TS ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): tablet 5704bdfe6c02437c8e9c8c3deaadb9e3 (table rebalance_test_table_6 [id=cc14910d53d74d46a1a9e2001d10f204]) successfully deleted
I20240502 20:46:31.873286 31434 ts_tablet_manager.cc:902] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:31.874428 31434 tablet_copy_client.cc:308] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:31.876312 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:35399)
I20240502 20:46:31.876458 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:35399: session id = 1f40a11333e44ea0a34b6bc7618af522-62c71a33fbcc4cfdbf0065a3fc9dfeba
I20240502 20:46:31.882225 27249 tablet_copy_source_session.cc:205] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:31.890465 31434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:31.895026 31434 tablet_copy_client.cc:783] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:31.895437 31434 tablet_copy_client.cc:647] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:31.910766 31434 tablet_copy_client.cc:523] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:31.912576 31434 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:31.919515 31497 ts_tablet_manager.cc:902] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:31.920315 31497 tablet_copy_client.cc:308] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:31.922641 27381 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 14a7bead40fb4eb0bec6b00804b52595 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:44893)
I20240502 20:46:31.922796 27381 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 14a7bead40fb4eb0bec6b00804b52595 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:44893: session id = 6706d1c0946d42a9bae62d719e1e5c31-14a7bead40fb4eb0bec6b00804b52595
I20240502 20:46:31.923849 27381 tablet_copy_source_session.cc:205] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:31.926263 31497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a7bead40fb4eb0bec6b00804b52595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:31.930090 31497 tablet_copy_client.cc:783] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:31.930311 31497 tablet_copy_client.cc:647] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:31.932791 31497 tablet_copy_client.cc:523] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:31.934485 31497 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:31.943240 31506 ts_tablet_manager.cc:902] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:31.944793 31506 tablet_copy_client.cc:308] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:31.945488 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet a547561343304137a044ce11517f6007 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:35399)
I20240502 20:46:31.945631 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet a547561343304137a044ce11517f6007 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:35399: session id = 1f40a11333e44ea0a34b6bc7618af522-a547561343304137a044ce11517f6007
I20240502 20:46:31.946758 27249 tablet_copy_source_session.cc:205] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:31.961591 31506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a547561343304137a044ce11517f6007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:31.967008 31506 tablet_copy_client.cc:783] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:31.967408 31506 tablet_copy_client.cc:647] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:31.971143 31506 tablet_copy_client.cc:523] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:31.974406 31506 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:31.981176 31497 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:31.981597 31497 tablet_bootstrap.cc:492] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:31.981819 31497 ts_tablet_manager.cc:1367] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.047s	user 0.031s	sys 0.000s
I20240502 20:46:31.982398 31497 raft_consensus.cc:348] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:31.982568 31497 raft_consensus.cc:729] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:31.982679 31497 consensus_queue.cc:260] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:31.983013 31497 ts_tablet_manager.cc:1398] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:31.983644 27381 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-14a7bead40fb4eb0bec6b00804b52595 received from {username='slave'} at 127.19.66.136:44893
I20240502 20:46:31.983752 27381 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-14a7bead40fb4eb0bec6b00804b52595 on tablet 14a7bead40fb4eb0bec6b00804b52595 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:31.996263 31434 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:31.996857 31434 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:31.997176 31434 ts_tablet_manager.cc:1367] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.085s	user 0.030s	sys 0.002s
I20240502 20:46:31.997794 31434 raft_consensus.cc:348] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.998021 31434 raft_consensus.cc:729] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:31.998195 31434 consensus_queue.cc:260] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:31.998742 31434 ts_tablet_manager.cc:1398] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:46:31.999532 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-62c71a33fbcc4cfdbf0065a3fc9dfeba received from {username='slave'} at 127.19.66.133:35399
I20240502 20:46:31.999637 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-62c71a33fbcc4cfdbf0065a3fc9dfeba on tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:32.031723 31566 ts_tablet_manager.cc:902] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:32.032399 31566 tablet_copy_client.cc:308] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:32.033016 27381 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet f00d12e425284c36ae8baed56c7cfaf6 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:53843)
I20240502 20:46:32.033192 27381 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet f00d12e425284c36ae8baed56c7cfaf6 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:53843: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-f00d12e425284c36ae8baed56c7cfaf6
I20240502 20:46:32.033486 31506 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:32.034070 31506 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap complete.
I20240502 20:46:32.034287 27381 tablet_copy_source_session.cc:205] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:32.034435 31506 ts_tablet_manager.cc:1367] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522: Time spent bootstrapping tablet: real 0.060s	user 0.033s	sys 0.004s
I20240502 20:46:32.038595 31506 raft_consensus.cc:348] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:32.038961 31506 raft_consensus.cc:729] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f40a11333e44ea0a34b6bc7618af522, State: Initialized, Role: LEARNER
I20240502 20:46:32.039198 31506 consensus_queue.cc:260] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:32.040660 31506 ts_tablet_manager.cc:1398] T a547561343304137a044ce11517f6007 P 1f40a11333e44ea0a34b6bc7618af522: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.000s
I20240502 20:46:32.041373 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1f40a11333e44ea0a34b6bc7618af522-a547561343304137a044ce11517f6007 received from {username='slave'} at 127.19.66.133:35399
I20240502 20:46:32.041502 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-a547561343304137a044ce11517f6007 on tablet a547561343304137a044ce11517f6007 with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:32.042419 31566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00d12e425284c36ae8baed56c7cfaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:32.055707 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.056784 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.062657 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.062973 28825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.079034 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:32.085438 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.085572 28825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.087872 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:32.112658 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:32.122596 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:32.123261 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:32.134532 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:32.164597 31506 ts_tablet_manager.cc:902] T 40df15a82a624913b3b712687d999134 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:32.165299 31506 tablet_copy_client.cc:308] T 40df15a82a624913b3b712687d999134 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:32.176476 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:32.180702 31434 ts_tablet_manager.cc:902] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:32.181147 31566 tablet_copy_client.cc:783] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.181483 31566 tablet_copy_client.cc:647] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:32.186720 31566 tablet_copy_client.cc:523] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:32.189998 31434 tablet_copy_client.cc:308] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:32.190979 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.201356 27381 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 40df15a82a624913b3b712687d999134 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:35193)
I20240502 20:46:32.201686 27381 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 40df15a82a624913b3b712687d999134 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:35193: session id = 1f40a11333e44ea0a34b6bc7618af522-40df15a82a624913b3b712687d999134
I20240502 20:46:32.203207 27381 tablet_copy_source_session.cc:205] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:32.203639 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 6a8ce1df86914b2caf2d083d97eda4e6 from peer 1f40a11333e44ea0a34b6bc7618af522 ({username='slave'} at 127.19.66.133:35193)
I20240502 20:46:32.203756 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 6a8ce1df86914b2caf2d083d97eda4e6 from peer 1f40a11333e44ea0a34b6bc7618af522 at {username='slave'} at 127.19.66.133:35193: session id = 1f40a11333e44ea0a34b6bc7618af522-6a8ce1df86914b2caf2d083d97eda4e6
I20240502 20:46:32.204632 31566 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:32.204795 27382 tablet_copy_source_session.cc:205] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:32.208838 31506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40df15a82a624913b3b712687d999134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:32.209571 31434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8ce1df86914b2caf2d083d97eda4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:32.212458 31434 tablet_copy_client.cc:783] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.212646 31434 tablet_copy_client.cc:647] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:32.218420 31506 tablet_copy_client.cc:783] T 40df15a82a624913b3b712687d999134 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.219663 31506 tablet_copy_client.cc:647] T 40df15a82a624913b3b712687d999134 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:32.220458 31614 ts_tablet_manager.cc:902] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:32.221155 31614 tablet_copy_client.cc:308] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:32.222215 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet bf189c19164442e5af0827f1fb7dc085 from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:49537)
I20240502 20:46:32.222539 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet bf189c19164442e5af0827f1fb7dc085 from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:49537: session id = 1bc8a142cfc7487bacc573afb0a92b74-bf189c19164442e5af0827f1fb7dc085
I20240502 20:46:32.222872 31506 tablet_copy_client.cc:523] T 40df15a82a624913b3b712687d999134 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:32.223541 27249 tablet_copy_source_session.cc:205] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:32.224610 31614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf189c19164442e5af0827f1fb7dc085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:32.224738 31506 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:32.227587 31614 tablet_copy_client.cc:783] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.227800 31614 tablet_copy_client.cc:647] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:32.229239 31614 tablet_copy_client.cc:523] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:32.231345 31614 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:32.239408 31576 ts_tablet_manager.cc:902] T a547561343304137a044ce11517f6007 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:32.239956 31576 tablet_copy_client.cc:308] T a547561343304137a044ce11517f6007 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:32.240854 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet a547561343304137a044ce11517f6007 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:39121)
I20240502 20:46:32.240974 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet a547561343304137a044ce11517f6007 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:39121: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-a547561343304137a044ce11517f6007
I20240502 20:46:32.241204 31619 ts_tablet_manager.cc:902] T f00d12e425284c36ae8baed56c7cfaf6 P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:32.242087 27249 tablet_copy_source_session.cc:205] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:32.243432 31612 ts_tablet_manager.cc:902] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:32.243666 28848 raft_consensus.cc:1203] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 1f40a11333e44ea0a34b6bc7618af522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
F20240502 20:46:32.241410 31619 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.queue_time_us, tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.queue_time_us) (tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_cpu_time_us, tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_cpu_time_us) (tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_wall_time_us, tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_wall_time_us) (tablet-download-pool-367543df62e048268908eb0d84c14876.queue_time_us, tablet-download-pool-367543df62e048268908eb0d84c14876.queue_time_us) (tablet-download-pool-367543df62e048268908eb0d84c14876.run_cpu_time_us, tablet-download-pool-367543df62e048268908eb0d84c14876.run_cpu_time_us) (tablet-download-pool-367543df62e048268908eb0d84c14876.run_wall_time_us, tablet-download-pool-367543df62e048268908eb0d84c14876.run_wall_time_us) (tablet-download-pool-40df15a82a624913b3b712687d999134.queue_time_us, tablet-download-pool-40df15a82a624913b3b712687d999134.queue_time_us) (tablet-download-pool-40df15a82a624913b3b712687d999134.run_cpu_time_us, tablet-download-pool-40df15a82a624913b3b712687d999134.run_cpu_time_us) (tablet-download-pool-40df15a82a624913b3b712687d999134.run_wall_time_us, tablet-download-pool-40df15a82a624913b3b712687d999134.run_wall_time_us) (tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.queue_time_us, tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.queue_time_us) (tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.run_cpu_time_us, tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.run_cpu_time_us) (tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.run_wall_time_us, tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.run_wall_time_us) (tablet-download-pool-62c71a33fbcc4cfdbf0065a3fc9dfeba.queue_time_us, tablet-download-pool-62c71a33fbcc4cfdbf0065a3fc9dfeba.queue_time_us) (tablet-download-pool-62c71a33fbcc4cfdbf0065a3fc9dfeba.run_cpu_time_us, tablet-download-pool-62c71a33fbcc4cfdbf0065a3fc9dfeba.run_cpu_time_us) (tablet-download-pool-62c71a33fbcc4cfdbf0065a3fc9dfeba.run_wall_time_us, tablet-download-pool-62c71a33fbcc4cfdbf0065a3fc9dfeba.run_wall_time_us) (tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.queue_time_us, tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.queue_time_us) (tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.run_cpu_time_us, tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.run_cpu_time_us) (tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.run_wall_time_us, tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.run_wall_time_us) (tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.queue_time_us, tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.queue_time_us) (tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_cpu_time_us, tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_cpu_time_us) (tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_wall_time_us, tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_wall_time_us) (tablet-download-pool-a547561343304137a044ce11517f6007.queue_time_us, tablet-download-pool-a547561343304137a044ce11517f6007.queue_time_us) (tablet-download-pool-a547561343304137a044ce11517f6007.run_cpu_time_us, tablet-download-pool-a547561343304137a044ce11517f6007.run_cpu_time_us) (tablet-download-pool-a547561343304137a044ce11517f6007.run_wall_time_us, tablet-download-pool-a547561343304137a044ce11517f6007.run_wall_time_us) (tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.queue_time_us, tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.queue_time_us) (tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.run_cpu_time_us, tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.run_cpu_time_us) (tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.run_wall_time_us, tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.run_wall_time_us) (tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.queue_time_us, tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.queue_time_us) (tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.run_cpu_time_us, tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.run_cpu_time_us) (tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.run_wall_time_us, tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.run_wall_time_us) (tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.queue_time_us, tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.queue_time_us) (tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_cpu_time_us, tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_cpu_time_us) (tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_wall_time_us, tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_wall_time_us) (tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.queue_time_us, tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.queue_time_us) (tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.run_cpu_time_us, tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.run_cpu_time_us) (tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.run_wall_time_us, tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.run_wall_time_us) (tablet-download-pool-f00c4b7954564820875a25da5834416a.queue_time_us, tablet-download-pool-f00c4b7954564820875a25da5834416a.queue_time_us) (tablet-download-pool-f00c4b7954564820875a25da5834416a.run_cpu_time_us, tablet-download-pool-f00c4b7954564820875a25da5834416a.run_cpu_time_us) (tablet-download-pool-f00c4b7954564820875a25da5834416a.run_wall_time_us, tablet-download-pool-f00c4b7954564820875a25da5834416a.run_wall_time_us) (tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.queue_time_us, tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.queue_time_us) (tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.queue_time_us, tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.queue_time_us) (tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_cpu_time_us, tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_cpu_time_us) (tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_wall_time_us, tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 20:46:32.245780 31612 tablet_copy_client.cc:308] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:32.246987 31434 tablet_copy_client.cc:523] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1f40a11333e44ea0a34b6bc7618af522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:32.248000 31497 ts_tablet_manager.cc:902] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:32.248572 31497 tablet_copy_client.cc:308] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:32.249207 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet ff9fef57f6eb490da536170a02b96212 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:58553)
I20240502 20:46:32.249336 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet ff9fef57f6eb490da536170a02b96212 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:58553: session id = 6706d1c0946d42a9bae62d719e1e5c31-ff9fef57f6eb490da536170a02b96212
I20240502 20:46:32.250478 27112 tablet_copy_source_session.cc:205] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:32.251046 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.257365 31636 ts_tablet_manager.cc:902] T 40df15a82a624913b3b712687d999134 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:32.259128 31636 tablet_copy_client.cc:308] T 40df15a82a624913b3b712687d999134 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:32.260030 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 40df15a82a624913b3b712687d999134 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:53843)
I20240502 20:46:32.260161 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 40df15a82a624913b3b712687d999134 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:53843: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-40df15a82a624913b3b712687d999134
I20240502 20:46:32.261709 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet ff9fef57f6eb490da536170a02b96212 from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:37781)
I20240502 20:46:32.261847 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet ff9fef57f6eb490da536170a02b96212 from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:37781: session id = 297de823852d4a8da44791ab03131cd3-ff9fef57f6eb490da536170a02b96212
I20240502 20:46:32.261993 31497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9fef57f6eb490da536170a02b96212. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:32.262897 27112 tablet_copy_source_session.cc:205] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:32.265024 27382 tablet_copy_source_session.cc:205] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:32.266415 31576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a547561343304137a044ce11517f6007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:32.266870 31612 ts_tablet_manager.cc:1875] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:46:32.269445 31566 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:32.269894 31566 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap complete.
I20240502 20:46:32.270119 31566 ts_tablet_manager.cc:1367] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent bootstrapping tablet: real 0.066s	user 0.029s	sys 0.004s
I20240502 20:46:32.270773 31566 raft_consensus.cc:348] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:32.270951 31566 raft_consensus.cc:729] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1af1e87d6ff2448d9502de9d7c2b8ef3, State: Initialized, Role: LEARNER
I20240502 20:46:32.271080 31566 consensus_queue.cc:260] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:32.271412 31434 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1f40a11333e44ea0a34b6bc7618af522: Bootstrap starting.
I20240502 20:46:32.271477 31566 ts_tablet_manager.cc:1398] T f00d12e425284c36ae8baed56c7cfaf6 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:32.272818 31612 ts_tablet_manager.cc:1888] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
    @     0x7f7d708cbdcd  google::LogMessage::Fail() at ??:0
I20240502 20:46:32.273609 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-f00d12e425284c36ae8baed56c7cfaf6 received from {username='slave'} at 127.19.66.134:53843
I20240502 20:46:32.273717 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-f00d12e425284c36ae8baed56c7cfaf6 on tablet f00d12e425284c36ae8baed56c7cfaf6 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:32.275213 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:32.275359 31612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9fef57f6eb490da536170a02b96212. 1 dirs total, 0 dirs full, 0 dirs failed
    @     0x7f7d708cfb93  google::LogMessage::SendToLog() at ??:0
I20240502 20:46:32.276234 31566 ts_tablet_manager.cc:902] T d1b9ff3c921d474fa5ad106d1c17ab00 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:32.277221 31566 tablet_copy_client.cc:308] T d1b9ff3c921d474fa5ad106d1c17ab00 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:32.277668 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet d1b9ff3c921d474fa5ad106d1c17ab00 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 ({username='slave'} at 127.19.66.134:40783)
I20240502 20:46:32.277776 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet d1b9ff3c921d474fa5ad106d1c17ab00 from peer 1af1e87d6ff2448d9502de9d7c2b8ef3 at {username='slave'} at 127.19.66.134:40783: session id = 1af1e87d6ff2448d9502de9d7c2b8ef3-d1b9ff3c921d474fa5ad106d1c17ab00
I20240502 20:46:32.278788 27112 tablet_copy_source_session.cc:205] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:32.278913 31636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40df15a82a624913b3b712687d999134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:32.280263 31566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b9ff3c921d474fa5ad106d1c17ab00. 1 dirs total, 0 dirs full, 0 dirs failed
    @     0x7f7d708cb7cc  google::LogMessage::Flush() at ??:0
I20240502 20:46:32.283264 31636 tablet_copy_client.cc:783] T 40df15a82a624913b3b712687d999134 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.283221 31612 tablet_copy_client.cc:783] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.283440 31636 tablet_copy_client.cc:647] T 40df15a82a624913b3b712687d999134 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:32.283684 31566 tablet_copy_client.cc:783] T d1b9ff3c921d474fa5ad106d1c17ab00 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.283998 31566 tablet_copy_client.cc:647] T d1b9ff3c921d474fa5ad106d1c17ab00 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:32.284200 31576 tablet_copy_client.cc:783] T a547561343304137a044ce11517f6007 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.284348 31576 tablet_copy_client.cc:647] T a547561343304137a044ce11517f6007 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Starting download of 1 WAL segments...
    @     0x7f7d708ccf59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 20:46:32.289598 31576 tablet_copy_client.cc:523] T a547561343304137a044ce11517f6007 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:32.291714 31576 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
    @     0x7f7d71d35044  kudu::TraceMetrics::InternName() at ??:0
I20240502 20:46:32.293331 31497 tablet_copy_client.cc:783] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.293540 31497 tablet_copy_client.cc:647] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
    @     0x7f7d71d19f98  kudu::ThreadPool::ThreadPool() at ??:0
I20240502 20:46:32.297148 31497 tablet_copy_client.cc:523] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
    @     0x7f7d71d17efc  kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 20:46:32.303676 31612 tablet_copy_client.cc:647] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:32.304276 28962 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.305336 31654 ts_tablet_manager.cc:902] T cd1b1316da2d47938164ac5b75f11bae P 1f40a11333e44ea0a34b6bc7618af522: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
    @     0x7f7d748ca77a  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 20:46:32.305851 31663 ts_tablet_manager.cc:902] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:32.306497 31663 tablet_copy_client.cc:308] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:32.306581 31497 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:32.307075 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet a906569ab3b14520b4876bc8d5450f7d from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:49537)
I20240502 20:46:32.307197 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet a906569ab3b14520b4876bc8d5450f7d from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:49537: session id = 1bc8a142cfc7487bacc573afb0a92b74-a906569ab3b14520b4876bc8d5450f7d
I20240502 20:46:32.308528 27249 tablet_copy_source_session.cc:205] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
    @     0x7f7d748d3a45  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 20:46:32.310530 31636 tablet_copy_client.cc:523] T 40df15a82a624913b3b712687d999134 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
    @     0x7f7d7495476f  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 20:46:32.320221 31636 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:32.320401 31612 tablet_copy_client.cc:523] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
    @     0x7f7d749684f6  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 20:46:32.323828 31612 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
    @     0x7f7d74953226  _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 20:46:32.326923 31663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a906569ab3b14520b4876bc8d5450f7d. 1 dirs total, 0 dirs full, 0 dirs failed
    @     0x7f7d74962c0a  _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x56543d75775c  std::function<>::operator()() at ??:0
    @     0x7f7d71d1ce06  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f7d71d1d6ef  _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
    @     0x7f7d71d1f03b  _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x56543d75775c  std::function<>::operator()() at ??:0
I20240502 20:46:32.340878 31646 ts_tablet_manager.cc:902] T cd1b1316da2d47938164ac5b75f11bae P 1af1e87d6ff2448d9502de9d7c2b8ef3: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
F20240502 20:46:32.341158 31646 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.queue_time_us, tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.queue_time_us) (tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_cpu_time_us, tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_cpu_time_us) (tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_wall_time_us, tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_wall_time_us) (tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.queue_time_us, tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.queue_time_us) (tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.run_cpu_time_us, tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.run_cpu_time_us) (tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.run_wall_time_us, tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.run_wall_time_us) (tablet-download-pool-367543df62e048268908eb0d84c14876.queue_time_us, tablet-download-pool-367543df62e048268908eb0d84c14876.queue_time_us) (tablet-download-pool-367543df62e048268908eb0d84c14876.run_cpu_time_us, tablet-download-pool-367543df62e048268908eb0d84c14876.run_cpu_time_us) (tablet-download-pool-367543df62e048268908eb0d84c14876.run_wall_time_us, tablet-download-pool-367543df62e048268908eb0d84c14876.run_wall_time_us) (tablet-download-pool-40df15a82a624913b3b712687d999134.queue_time_us, tablet-download-pool-40df15a82a624913b3b712687d999134.queue_time_us) (tablet-download-pool-40df15a82a624913b3b712687d999134.run_cpu_time_us, tablet-download-pool-40df15a82a624913b3b712687d999134.run_cpu_time_us) (tablet-download-pool-40df15a82a624913b3b712687d999134.run_wall_time_us, tablet-download-pool-40df15a82a624913b3b712687d999134.run_wall_time_us) (tablet-download-pool-843a26fc48844bfd82911b6a27833023.queue_time_us, tablet-download-pool-843a26fc48844bfd82911b6a27833023.queue_time_us) (tablet-download-pool-843a26fc48844bfd82911b6a27833023.run_cpu_time_us, tablet-download-pool-843a26fc48844bfd82911b6a27833023.run_cpu_time_us) (tablet-download-pool-843a26fc48844bfd82911b6a27833023.run_wall_time_us, tablet-download-pool-843a26fc48844bfd82911b6a27833023.run_wall_time_us) (tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.queue_time_us, tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.queue_time_us) (tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_cpu_time_us, tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_cpu_time_us) (tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_wall_time_us, tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_wall_time_us) (tablet-download-pool-a2b61909194d488682afd91c0f796405.queue_time_us, tablet-download-pool-a2b61909194d488682afd91c0f796405.queue_time_us) (tablet-download-pool-a2b61909194d488682afd91c0f796405.run_cpu_time_us, tablet-download-pool-a2b61909194d488682afd91c0f796405.run_cpu_time_us) (tablet-download-pool-a2b61909194d488682afd91c0f796405.run_wall_time_us, tablet-download-pool-a2b61909194d488682afd91c0f796405.run_wall_time_us) (tablet-download-pool-a547561343304137a044ce11517f6007.queue_time_us, tablet-download-pool-a547561343304137a044ce11517f6007.queue_time_us) (tablet-download-pool-a547561343304137a044ce11517f6007.run_cpu_time_us, tablet-download-pool-a547561343304137a044ce11517f6007.run_cpu_time_us) (tablet-download-pool-a547561343304137a044ce11517f6007.run_wall_time_us, tablet-download-pool-a547561343304137a044ce11517f6007.run_wall_time_us) (tablet-download-pool-cd1b1316da2d47938164ac5b75f11bae.queue_time_us, tablet-download-pool-cd1b1316da2d47938164ac5b75f11bae.queue_time_us) (tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.queue_time_us, tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.queue_time_us) (tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.run_cpu_time_us, tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.run_cpu_time_us) (tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.run_wall_time_us, tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.run_wall_time_us) (tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.queue_time_us, tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.queue_time_us) (tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.run_cpu_time_us, tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.run_cpu_time_us) (tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.run_wall_time_us, tablet-download-pool-d44f271f166b4c288b76689a3e3764d3.run_wall_time_us) (tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.queue_time_us, tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.queue_time_us) (tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.run_cpu_time_us, tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.run_cpu_time_us) (tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.run_wall_time_us, tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.run_wall_time_us) (tablet-download-pool-f1545142b0bd40859bf33bf8686002ae.queue_time_us, tablet-download-pool-f1545142b0bd40859bf33bf8686002ae.queue_time_us) (tablet-download-pool-f1545142b0bd40859bf33bf8686002ae.run_cpu_time_us, tablet-download-pool-f1545142b0bd40859bf33bf8686002ae.run_cpu_time_us) (tablet-download-pool-f1545142b0bd40859bf33bf8686002ae.run_wall_time_us, tablet-download-pool-f1545142b0bd40859bf33bf8686002ae.run_wall_time_us) (tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.queue_time_us, tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.queue_time_us) (tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_cpu_time_us, tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_cpu_time_us) (tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_wall_time_us, tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_wall_time_us) (tablet-download-pool-ff9fef57f6eb490da536170a02b96212.queue_time_us, tablet-download-pool-ff9fef57f6eb490da536170a02b96212.queue_time_us) (tablet-download-pool-ff9fef57f6eb490da536170a02b96212.run_cpu_time_us, tablet-download-pool-ff9fef57f6eb490da536170a02b96212.run_cpu_time_us) (tablet-download-pool-ff9fef57f6eb490da536170a02b96212.run_wall_time_us, tablet-download-pool-ff9fef57f6eb490da536170a02b96212.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7f7d71d0c807  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f7996e6edcd  google::LogMessage::Fail() at ??:0
    @     0x7f7d7254a6db  start_thread at ??:0
    @     0x7f7996e72b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f7d6fa8271f  clone at ??:0
    @     0x7f7996e6e7cc  google::LogMessage::Flush() at ??:0
    @     0x7f7996e6ff59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 20:46:32.359647 29252 raft_consensus.cc:1203] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
    @     0x7f79982d8044  kudu::TraceMetrics::InternName() at ??:0
    @     0x7f79982bcf98  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7f79982baefc  kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 20:46:32.372829 31614 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:32.373364 31614 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:32.373639 31614 ts_tablet_manager.cc:1367] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.142s	user 0.053s	sys 0.006s
I20240502 20:46:32.374301 31614 raft_consensus.cc:348] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:32.374491 31614 raft_consensus.cc:729] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:32.374644 31614 consensus_queue.cc:260] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:32.375133 31614 ts_tablet_manager.cc:1398] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240502 20:46:32.376166 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-bf189c19164442e5af0827f1fb7dc085 received from {username='slave'} at 127.19.66.135:49537
I20240502 20:46:32.376253 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-bf189c19164442e5af0827f1fb7dc085 on tablet bf189c19164442e5af0827f1fb7dc085 with peer 1bc8a142cfc7487bacc573afb0a92b74
    @     0x7f799ae6d77a  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7f799ae76a45  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7f799aef776f  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 20:46:32.385587 31648 ts_tablet_manager.cc:902] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:46:32.386140 31648 tablet_copy_client.cc:308] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:32.386778 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 8deaa1f751034c15acd6ccaad37536c6 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:44893)
I20240502 20:46:32.386906 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 8deaa1f751034c15acd6ccaad37536c6 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:44893: session id = 6706d1c0946d42a9bae62d719e1e5c31-8deaa1f751034c15acd6ccaad37536c6
I20240502 20:46:32.388295 27382 tablet_copy_source_session.cc:205] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:32.393579 31648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8deaa1f751034c15acd6ccaad37536c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:32.394425 31566 tablet_copy_client.cc:523] T d1b9ff3c921d474fa5ad106d1c17ab00 P 1af1e87d6ff2448d9502de9d7c2b8ef3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:32.396234 31566 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P 1af1e87d6ff2448d9502de9d7c2b8ef3: Bootstrap starting.
I20240502 20:46:32.397245 31648 tablet_copy_client.cc:783] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.397423 31648 tablet_copy_client.cc:647] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
    @     0x7f799af0b4f6  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7f799aef6226  _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 20:46:32.411041 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.412058 31663 tablet_copy_client.cc:783] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:32.412293 31663 tablet_copy_client.cc:647] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
    @     0x7f799af05c0a  _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x55618367375c  std::function<>::operator()() at ??:0
    @     0x7f79982bfe06  kudu::ThreadPool::DispatchThread() at ??:0
I20240502 20:46:32.435652 27359 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:32.438082 27226 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:32.438874 29117 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:32.438854 31211 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:32.440066 31210 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:32.444584 31663 tablet_copy_client.cc:523] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:32.446869 31663 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
    @     0x7f79982c06ef  _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240502 20:46:32.451220 31210 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:32.452409 31497 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:32.452859 31497 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:32.453085 31497 ts_tablet_manager.cc:1367] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.147s	user 0.037s	sys 0.005s
    @     0x7f79982c203b  _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x55618367375c  std::function<>::operator()() at ??:0
I20240502 20:46:32.453912 29117 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:32.454363 31497 raft_consensus.cc:348] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:46:32.454546 31497 raft_consensus.cc:729] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:32.454680 31497 consensus_queue.cc:260] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:46:32.455039 31497 ts_tablet_manager.cc:1398] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 20:46:32.455950 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-ff9fef57f6eb490da536170a02b96212 received from {username='slave'} at 127.19.66.136:58553
I20240502 20:46:32.456045 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-ff9fef57f6eb490da536170a02b96212 on tablet ff9fef57f6eb490da536170a02b96212 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:32.457542 26805 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
    @     0x7f79982af807  kudu::Thread::SuperviseThread() at ??:0
I20240502 20:46:32.459364 27226 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
    @     0x7f7998aed6db  start_thread at ??:0
I20240502 20:46:32.471168 31612 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:32.471589 31612 tablet_bootstrap.cc:492] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:46:32.471848 31612 ts_tablet_manager.cc:1367] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.148s	user 0.042s	sys 0.000s
    @     0x7f799602571f  clone at ??:0
I20240502 20:46:32.472576 31612 raft_consensus.cc:348] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:46:32.472759 31612 raft_consensus.cc:729] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:46:32.473615 31612 consensus_queue.cc:260] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:46:32.474054 31612 ts_tablet_manager.cc:1398] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:46:32.475033 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-ff9fef57f6eb490da536170a02b96212 received from {username='slave'} at 127.19.66.129:37781
I20240502 20:46:32.475128 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 297de823852d4a8da44791ab03131cd3-ff9fef57f6eb490da536170a02b96212 on tablet ff9fef57f6eb490da536170a02b96212 with peer 297de823852d4a8da44791ab03131cd3
I20240502 20:46:32.479558 31648 tablet_copy_client.cc:523] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:32.483484 31648 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:32.487892 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:32.498607 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 20:46:32.507887 29230 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 20:46:32.537272 27226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a547561343304137a044ce11517f6007"
dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34188
I20240502 20:46:32.537494 27226 raft_consensus.cc:595] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:32.546986 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:32.557188 27207 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 20:46:32.569041 26960 consensus_queue.cc:237] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:32.571907 27359 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:32.572870 30908 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:32.575680 30908 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:32.580113 26806 catalog_manager.cc:5484] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:32.581663 27359 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
W20240502 20:46:32.597405 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2b61909194d488682afd91c0f796405. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:32.608866 27359 consensus_queue.cc:237] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:32.612082 26960 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 20:46:32.612643 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a8ce1df86914b2caf2d083d97eda4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:32.612879 31211 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:32.613860 29117 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:32.614637 31211 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:32.618717 31211 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:32.622035 29117 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:32.626157 26960 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:32.629583 26806 catalog_manager.cc:5484] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 8 to 9, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:32.631968 30248 raft_consensus.cc:979] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8: : Instructing follower 6706d1c0946d42a9bae62d719e1e5c31 to start an election
I20240502 20:46:32.632084 30248 raft_consensus.cc:1067] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Signalling peer 6706d1c0946d42a9bae62d719e1e5c31 to start an election
I20240502 20:46:32.637260 29252 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a547561343304137a044ce11517f6007"
dest_uuid: "6706d1c0946d42a9bae62d719e1e5c31"
 from {username='slave'} at 127.19.66.131:37143
I20240502 20:46:32.637495 29252 raft_consensus.cc:482] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:32.637614 29252 raft_consensus.cc:3037] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:32.638917 29252 raft_consensus.cc:504] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:32.641402 27226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a547561343304137a044ce11517f6007" candidate_uuid: "6706d1c0946d42a9bae62d719e1e5c31" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:32.641589 27226 raft_consensus.cc:3032] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:32.641662 27226 raft_consensus.cc:729] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Running, Role: LEADER
I20240502 20:46:32.641893 27226 consensus_queue.cc:260] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:32.642136 27226 raft_consensus.cc:3037] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:32.643556 27226 raft_consensus.cc:2454] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6706d1c0946d42a9bae62d719e1e5c31 in term 2.
I20240502 20:46:32.645237 29252 leader_election.cc:288] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [CANDIDATE]: Term 2 election: Requested vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:32.644994 29186 leader_election.cc:302] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6706d1c0946d42a9bae62d719e1e5c31, c107c43e538d48f58625c4320f0aa4b8; no voters: 
I20240502 20:46:32.646765 30906 raft_consensus.cc:2781] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:32.647003 30906 raft_consensus.cc:686] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [term 2 LEADER]: Becoming Leader. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Running, Role: LEADER
I20240502 20:46:32.647403 30906 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:32.649850 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a547561343304137a044ce11517f6007" candidate_uuid: "6706d1c0946d42a9bae62d719e1e5c31" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:32.650075 27093 raft_consensus.cc:3037] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:32.651610 27093 raft_consensus.cc:2454] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6706d1c0946d42a9bae62d719e1e5c31 in term 2.
I20240502 20:46:32.666371 27226 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:46:32.668781 26806 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: term changed from 1 to 2, leader changed from c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) to 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136). New cstate: current_term: 2 leader_uuid: "6706d1c0946d42a9bae62d719e1e5c31" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:32.670514 31648 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:32.670895 31648 tablet_bootstrap.cc:492] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:32.671110 31648 ts_tablet_manager.cc:1367] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.188s	user 0.028s	sys 0.005s
I20240502 20:46:32.671650 31648 raft_consensus.cc:348] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:32.671785 31648 raft_consensus.cc:729] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:32.676259 26960 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:32.676528 29117 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:32.709918 31663 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:32.677377 31180 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:32.776882 31663 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:32.777333 31663 ts_tablet_manager.cc:1367] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.331s	user 0.038s	sys 0.000s
I20240502 20:46:32.778244 31663 raft_consensus.cc:348] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:32.778448 31663 raft_consensus.cc:729] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:32.778578 31663 consensus_queue.cc:260] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:32.777051 31181 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20240502 20:46:32.779016 31663 ts_tablet_manager.cc:1398] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
W20240502 20:46:32.780531 27161 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): Couldn't send request to peer d1f3f8e444364d6faa42d651b5dbb997. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:32.671895 31648 consensus_queue.cc:260] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:32.832639 31648 ts_tablet_manager.cc:1398] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.161s	user 0.003s	sys 0.000s
I20240502 20:46:32.786124 26960 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:46:32.827948 26957 raft_consensus.cc:1203] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 20:46:32.839406 29252 raft_consensus.cc:1203] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20240502 20:46:32.854054 27024 connection.cc:535] client connection to 127.19.66.133:45949 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:32.854223 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:46:32.865485 26806 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 6 to 7, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
W20240502 20:46:32.875661 27024 consensus_peers.cc:485] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:32.875805 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:32.951819 26891 connection.cc:535] client connection to 127.19.66.134:46737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:32.952040 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:32.875866 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.000552 29304 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskF51BWD/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 20:46:25.574412 29306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:25.574867 29306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:25.580066 29306 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:46:27.128561 29306 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 20:46:27.959506 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.959631 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.959671 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.959697 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.959730 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.959767 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.959795 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.959830 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.959880 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 01e571f10ffe4d4c88601420e733e5c7: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.959919 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.959968 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.960003 29306 rebalancer_tool.cc:1205] No I20240502 20:46:33.024086 29230 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:32.829545 31749 raft_consensus.cc:1050] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:33.090049 31749 consensus_queue.cc:237] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:32.836584 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-8deaa1f751034c15acd6ccaad37536c6 received from {username='slave'} at 127.19.66.136:44893
I20240502 20:46:33.091492 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-8deaa1f751034c15acd6ccaad37536c6 on tablet 8deaa1f751034c15acd6ccaad37536c6 with peer 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:32.857245 27295 connection.cc:535] client connection to 127.19.66.133:45949 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:33.092334 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:46:33.092734 29252 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 20:46:33.092871 27295 connection.cc:698] client connection to 127.19.66.134:46737 send error: Network error: sendmsg error: Connection reset by peer (error 104)
I20240502 20:46:33.093786 31211 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:32.890738 27359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8deaa1f751034c15acd6ccaad37536c6"
dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38636
I20240502 20:46:33.094257 27359 raft_consensus.cc:595] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Received request to transfer leadership
W20240502 20:46:33.084900 29183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:33.128351 27342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.960050 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.960088 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.960124 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.960163 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.960202 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:28.000792 29306 rebalancer_tool.cc:1074] tablet 367543df62e048268908eb0d84c14876: 'c107c43e538d48f58625c4320f0aa4b8' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:28.030015 29306 rebalancer_tool.cc:1074] tablet 8deaa1f751034c15acd6ccaad37536c6: '297de823852d4a8da44791ab03131cd3' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:28.094383 29306 rebalancer_tool.cc:1074] tablet 6a8ce1df86914b2caf2d083d97eda4e6: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:28.174343 29306 rebalancer_tool.cc:1074] tablet a906569ab3b14520b4876bc8d5450f7d: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:28.244300 29306 rebalancer_tool.cc:1074] tablet fdd2311ec0b645f5a79098a32c6deba2: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:28.412935 29306 rebalancer_tool.cc:1074] tablet d44f271f166b4c288b76689a3e3764d3: '297de823852d4a8da44791ab03131cd3' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:28.581414 29306 rebalancer_tool.cc:1084] tablet 5704bdfe6c02437c8e9c8c3deaadb9e3: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1bc8a142cfc7487bacc573afb0a92b74' move ignored: Illegal state: tablet 5704bdfe6c02437c8e9c8c3deaadb9e3: replica 1bc8a142cfc7487bacc573afb0a92b74 is already present
I20240502 20:46:28.581553 29306 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:29.989866 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:29.989982 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:29.990039 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 01e571f10ffe4d4c88601420e733e5c7: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:29.990078 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:29.990109 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:29.990139 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:29.990172 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:29.990201 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:29.990231 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:29.990257 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:29.990283 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:29.990324 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:29.990361 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:29.990394 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:29.990420 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 01e571f10ffe4d4c88601420e733e5c7: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:29.990444 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:29.990473 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:46:30.116122 29306 rebalancer_tool.cc:1084] tablet f2dfc0bbbbca4cc28a97480007343424: 'd1f3f8e444364d6faa42d651b5dbb997' -> '6706d1c0946d42a9bae62d719e1e5c31' move ignored: Illegal state: tablet f2dfc0bbbbca4cc28a97480007343424: replica 6706d1c0946d42a9bae62d719e1e5c31 is already present
I20240502 20:46:30.116225 29306 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:31.404290 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:31.404436 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:31.404474 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:31.404556 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:31.404592 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:31.404629 29306 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:31.495690 29306 rebalancer_tool.cc:1074] tablet a547561343304137a044ce11517f6007: 'd1f3f8e444364d6faa42d651b5dbb997' -> '' move scheduled
I20240502 20:46:31.667874 29306 rebalancer_tool.cc:1074] tablet f00d12e425284c36ae8baed56c7cfaf6: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:31.778399 29306 rebalancer_tool.cc:1074] tablet bf189c19164442e5af0827f1fb7dc085: 'c107c43e538d48f58625c4320f0aa4b8' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:31.828058 29306 rebalancer_tool.cc:1074] tablet ff9fef57f6eb490da536170a02b96212: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '297de823852d4a8da44791ab03131cd3' move scheduled
I20240502 20:46:31.858991 29306 rebalancer_tool.cc:1084] tablet 40df15a82a624913b3b712687d999134: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move ignored: Illegal state: tablet 40df15a82a624913b3b712687d999134: replica 1af1e87d6ff2448d9502de9d7c2b8ef3 is already present
I20240502 20:46:31.859088 29306 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:32.103513 31521 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.19.66.190:43237, user_credentials={real_user=slave}} blocked reactor thread for 44473.3us
W20240502 20:46:32.169693 31521 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.19.66.190:43237, user_credentials={real_user=slave}} blocked reactor thread for 65731.6us
W20240502 20:46:32.285454 31521 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.19.66.190:43237, user_credentials={real_user=slave}} blocked reactor thread for 47358.3us
W20240502 20:46:32.852649 31440 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:46:32.337929 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.133:45949 (local address 127.0.0.1:47970)
0502 20:46:32.350217 (+ 12288us) negotiation.cc:107] Waiting for socket to connect
0502 20:46:32.350233 (+    16us) client_negotiation.cc:174] Beginning negotiation
0502 20:46:32.350319 (+    86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:46:32.357239 (+  6920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 20:46:32.357243 (+     4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 20:46:32.357274 (+    31us) client_negotiation.cc:189] Negotiated authn=SASL
0502 20:46:32.357560 (+   286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 20:46:32.357571 (+    11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 20:46:32.851619 (+494048us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":12209,"thread_start_us":82,"threads_started":1}
W20240502 20:46:32.854911 31440 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:46:32.853324 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.133:45949 (local address 127.0.0.1:47980)
0502 20:46:32.854732 (+  1408us) negotiation.cc:107] Waiting for socket to connect
0502 20:46:32.854807 (+    75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1297}
Illegal state: tablet server 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): unacceptable health status UNAVAILABLE

W20240502 20:46:32.780650 27161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.66.130:43307, user_credentials={real_user=slave}} blocked reactor thread for 102322us
I20240502 20:46:32.781636 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-a906569ab3b14520b4876bc8d5450f7d received from {username='slave'} at 127.19.66.135:49537
I20240502 20:46:33.145357 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-a906569ab3b14520b4876bc8d5450f7d on tablet a906569ab3b14520b4876bc8d5450f7d with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:32.782806 31181 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
W20240502 20:46:32.853794 27158 connection.cc:535] client connection to 127.19.66.133:45949 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:46:33.095796 27226 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 2 FOLLOWER]: Refusing update from remote peer 6706d1c0946d42a9bae62d719e1e5c31: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
W20240502 20:46:33.148626 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:33.149331 27158 connection.cc:698] client connection to 127.19.66.134:46737 send error: Network error: sendmsg error: Connection reset by peer (error 104)
I20240502 20:46:33.149222 31757 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:46:33.035992 27024 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.66.133:45949, user_credentials={real_user=slave}} blocked reactor thread for 160104us
W20240502 20:46:33.151495 27024 connection.cc:698] client connection to 127.19.66.134:46737 send error: Network error: sendmsg error: Connection reset by peer (error 104)
I20240502 20:46:33.010843 27093 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:32.783790 29117 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
W20240502 20:46:32.969209 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.157745 26891 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.66.134:46737, user_credentials={real_user=slave}} blocked reactor thread for 188545us
I20240502 20:46:33.023093 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:33.159477 27090 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Refusing update from remote peer 6706d1c0946d42a9bae62d719e1e5c31: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 20:46:33.095983 26957 raft_consensus.cc:1261] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:33.163987 29115 raft_consensus.cc:1203] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 20:46:33.164460 30906 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:46:33.169345 29183 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.171075 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.171214 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:33.171316 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.171743 27295 consensus_peers.cc:485] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.171815 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.171857 27295 consensus_peers.cc:485] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.176272 31648 ts_tablet_manager.cc:902] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
W20240502 20:46:33.176865 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.176970 27024 consensus_peers.cc:485] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.177029 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.178635 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.178684 29251 raft_consensus.cc:1203] T 8deaa1f751034c15acd6ccaad37536c6 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20240502 20:46:33.180507 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.182035 31648 tablet_copy_client.cc:308] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:46:33.183748 26960 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:33.184973 31210 consensus_queue.cc:1035] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:33.185832 31663 ts_tablet_manager.cc:902] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:33.186414 31663 tablet_copy_client.cc:308] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:38835
I20240502 20:46:33.187914 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 6a8ce1df86914b2caf2d083d97eda4e6 from peer 6706d1c0946d42a9bae62d719e1e5c31 ({username='slave'} at 127.19.66.136:44893)
I20240502 20:46:33.188050 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 6a8ce1df86914b2caf2d083d97eda4e6 from peer 6706d1c0946d42a9bae62d719e1e5c31 at {username='slave'} at 127.19.66.136:44893: session id = 6706d1c0946d42a9bae62d719e1e5c31-6a8ce1df86914b2caf2d083d97eda4e6
I20240502 20:46:33.188158 29117 raft_consensus.cc:1203] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 20:46:33.189352 27382 tablet_copy_source_session.cc:205] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:33.190145 30058 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:46:33.191440 27025 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f00c4b7954564820875a25da5834416a. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.193426 31648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8ce1df86914b2caf2d083d97eda4e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 20:46:33.194283 29183 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P 6706d1c0946d42a9bae62d719e1e5c31 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.199354 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:33.201318 31648 tablet_copy_client.cc:783] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:33.201673 31648 tablet_copy_client.cc:647] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:33.202039 27207 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:33.203833 31648 tablet_copy_client.cc:523] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:33.204012 29092 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 20:46:33.205098 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:46:33.209014 31648 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap starting.
I20240502 20:46:33.215603 31211 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:33.217932 26957 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:33.221364 29251 raft_consensus.cc:2932] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:33.223526 30058 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:33.225764 26960 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:33.227597 26805 catalog_manager.cc:5484] T 14a7bead40fb4eb0bec6b00804b52595 P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 7 to 8, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:46:33.235433 26801 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 8 to 9, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:46:33.235989 26980 tablet_copy_service.cc:140] P 297de823852d4a8da44791ab03131cd3: Received BeginTabletCopySession request for tablet a2b61909194d488682afd91c0f796405 from peer 1bc8a142cfc7487bacc573afb0a92b74 ({username='slave'} at 127.19.66.135:53571)
I20240502 20:46:33.236109 26980 tablet_copy_service.cc:161] P 297de823852d4a8da44791ab03131cd3: Beginning new tablet copy session on tablet a2b61909194d488682afd91c0f796405 from peer 1bc8a142cfc7487bacc573afb0a92b74 at {username='slave'} at 127.19.66.135:53571: session id = 1bc8a142cfc7487bacc573afb0a92b74-a2b61909194d488682afd91c0f796405
I20240502 20:46:33.237212 26980 tablet_copy_source_session.cc:205] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:33.238425 31663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b61909194d488682afd91c0f796405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:33.241474 31663 tablet_copy_client.cc:783] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:33.241681 31663 tablet_copy_client.cc:647] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Starting download of 1 WAL segments...
W20240502 20:46:33.247021 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:46:33.247062 31663 tablet_copy_client.cc:523] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 20:46:33.247236 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.247334 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.247398 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.247745 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:33.247879 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.247943 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.250416 31663 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap starting.
I20240502 20:46:33.269716 31201 raft_consensus.cc:1050] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 to VOTER
I20240502 20:46:33.270576 31201 consensus_queue.cc:237] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:46:33.274369 27359 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:33.276683 29250 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:33.278142 31921 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:33.281903 26960 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:33.282636 31201 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:33.284932 31921 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:46:33.286603 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.289273 27091 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:33.292515 27361 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 20:46:33.293283 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1b9ff3c921d474fa5ad106d1c17ab00. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.293283 31920 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:33.294466 29251 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:33.295397 31648 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:33.295871 31648 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31: Bootstrap complete.
I20240502 20:46:33.295878 31968 ts_tablet_manager.cc:902] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:33.296267 31648 ts_tablet_manager.cc:1367] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent bootstrapping tablet: real 0.087s	user 0.035s	sys 0.000s
I20240502 20:46:33.296957 31648 raft_consensus.cc:348] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:33.297322 31648 raft_consensus.cc:729] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6706d1c0946d42a9bae62d719e1e5c31, State: Initialized, Role: LEARNER
I20240502 20:46:33.297531 31648 consensus_queue.cc:260] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:33.298048 31920 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:33.299109 31648 ts_tablet_manager.cc:1398] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 20:46:33.299599 31968 tablet_copy_client.cc:308] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:46:33.299790 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-6a8ce1df86914b2caf2d083d97eda4e6 received from {username='slave'} at 127.19.66.136:44893
I20240502 20:46:33.299866 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 6706d1c0946d42a9bae62d719e1e5c31-6a8ce1df86914b2caf2d083d97eda4e6 on tablet 6a8ce1df86914b2caf2d083d97eda4e6 with peer 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:33.300416 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet 84484d65e2bb4e23ac621a39434ce382 from peer d1f3f8e444364d6faa42d651b5dbb997 ({username='slave'} at 127.19.66.130:60245)
I20240502 20:46:33.300539 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet 84484d65e2bb4e23ac621a39434ce382 from peer d1f3f8e444364d6faa42d651b5dbb997 at {username='slave'} at 127.19.66.130:60245: session id = d1f3f8e444364d6faa42d651b5dbb997-84484d65e2bb4e23ac621a39434ce382
I20240502 20:46:33.301653 27249 tablet_copy_source_session.cc:205] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 20:46:33.302412 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.305828 31968 ts_tablet_manager.cc:1875] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:46:33.308746 27361 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:33.311523 27091 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:46:33.311741 27297 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5586b9e60b544aed87c24f7b3ce0198c. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.312160 31749 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:33.312875 29252 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:33.313577 31749 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:33.314886 31921 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:33.315258 31968 ts_tablet_manager.cc:1888] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:46:33.315515 31968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84484d65e2bb4e23ac621a39434ce382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:33.315934 31749 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:33.319169 27091 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:33.319325 26801 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:33.319572 26812 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 7 to 8, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:33.318428 31968 tablet_copy_client.cc:783] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:33.321236 31968 tablet_copy_client.cc:647] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:33.325960 31201 raft_consensus.cc:1013] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: attempt to promote peer 297de823852d4a8da44791ab03131cd3: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 20:46:33.327124 31202 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.327600 27359 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:33.330276 29250 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.330580 29252 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:33.332664 27359 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.333321 31663 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:33.333683 31663 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74: Bootstrap complete.
I20240502 20:46:33.333899 31663 ts_tablet_manager.cc:1367] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent bootstrapping tablet: real 0.084s	user 0.033s	sys 0.000s
I20240502 20:46:33.334826 31663 raft_consensus.cc:348] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:33.334844 26960 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.335089 31663 raft_consensus.cc:729] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Initialized, Role: LEARNER
I20240502 20:46:33.335217 31663 consensus_queue.cc:260] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } }
I20240502 20:46:33.335211 26801 catalog_manager.cc:5484] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 8 to 9, 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.343436 31968 tablet_copy_client.cc:523] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:33.344087 31210 raft_consensus.cc:979] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b: : Instructing follower 1bc8a142cfc7487bacc573afb0a92b74 to start an election
I20240502 20:46:33.344199 31210 raft_consensus.cc:1067] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Signalling peer 1bc8a142cfc7487bacc573afb0a92b74 to start an election
I20240502 20:46:33.344977 29117 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8deaa1f751034c15acd6ccaad37536c6"
dest_uuid: "1bc8a142cfc7487bacc573afb0a92b74"
 from {username='slave'} at 127.19.66.132:41907
I20240502 20:46:33.345206 29117 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:33.345317 29117 raft_consensus.cc:3037] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:33.346038 31968 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:46:33.346467 29117 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:33.347976 31922 raft_consensus.cc:1050] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:46:33.348753 31922 consensus_queue.cc:237] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:33.349493 29117 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [CANDIDATE]: Term 2 election: Requested vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:33.350721 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8deaa1f751034c15acd6ccaad37536c6" candidate_uuid: "1bc8a142cfc7487bacc573afb0a92b74" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:33.350898 27359 raft_consensus.cc:3032] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:33.350955 27359 raft_consensus.cc:729] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Running, Role: LEADER
I20240502 20:46:33.351161 27359 consensus_queue.cc:260] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:33.351361 29252 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:33.351410 27359 raft_consensus.cc:3037] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:33.352175 31920 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:33.352643 27359 raft_consensus.cc:2454] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bc8a142cfc7487bacc573afb0a92b74 in term 2.
W20240502 20:46:33.350831 29048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:33.354595 29050 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bc8a142cfc7487bacc573afb0a92b74, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 
I20240502 20:46:33.354903 31680 raft_consensus.cc:2781] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 20:46:33.355088 31680 raft_consensus.cc:686] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [term 2 LEADER]: Becoming Leader. State: Replica: 1bc8a142cfc7487bacc573afb0a92b74, State: Running, Role: LEADER
I20240502 20:46:33.355407 31680 consensus_queue.cc:237] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:33.358544 27359 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:33.359083 31921 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:46:33.359764 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.360637 26809 catalog_manager.cc:5484] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: term changed from 1 to 2, leader changed from ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) to 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135). New cstate: current_term: 2 leader_uuid: "1bc8a142cfc7487bacc573afb0a92b74" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:33.362689 31202 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:46:33.366231 27359 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:46:33.367213 29252 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:46:33.369288 26812 catalog_manager.cc:5484] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:33.370286 29048 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:33.371016 26960 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:33.373158 31922 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:33.374795 26957 raft_consensus.cc:2932] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:46:33.389375 31663 ts_tablet_manager.cc:1398] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74: Time spent starting tablet: real 0.055s	user 0.001s	sys 0.000s
I20240502 20:46:33.390421 26980 tablet_copy_service.cc:342] P 297de823852d4a8da44791ab03131cd3: Request end of tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-a2b61909194d488682afd91c0f796405 received from {username='slave'} at 127.19.66.135:53571
I20240502 20:46:33.390563 26980 tablet_copy_service.cc:433] P 297de823852d4a8da44791ab03131cd3: ending tablet copy session 1bc8a142cfc7487bacc573afb0a92b74-a2b61909194d488682afd91c0f796405 on tablet a2b61909194d488682afd91c0f796405 with peer 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:33.402971 29301 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskF51BWD/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 20:46:25.648671 29307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:25.649051 29307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:25.654379 29307 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:46:27.079850 29307 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 20:46:27.498374 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.498575 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.498663 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.498736 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.498838 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.498885 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.498937 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 01e571f10ffe4d4c88601420e733e5c7: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.499029 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.499107 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.499159 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.499202 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.499240 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.522630 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.522774 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.522835 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:27.603845 29307 rebalancer_tool.cc:1074] tablet d1b9ff3c921d474fa5ad106d1c17ab00: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:27.667908 29307 rebalancer_tool.cc:1074] tablet ff9fef57f6eb490da536170a02b96212: '297de823852d4a8da44791ab03131cd3' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:27.856508 29307 rebalancer_tool.cc:1074] tablet f00c4b7954564820875a25da5834416a: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:27.943552 29307 rebalancer_tool.cc:1074] tablet 5cf737e20ca74db1b94ccff5d60133ee: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:28.009706 29307 rebalancer_tool.cc:1074] tablet 135a08f1f2ac4707bbacee81ac73a19e: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:28.084802 29307 rebalancer_tool.cc:1074] tablet 078a1591d9d74e31a0407aeab76911b4: '297de823852d4a8da44791ab03131cd3' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:28.211618 29307 rebalancer_tool.cc:1074] tablet d44f271f166b4c288b76689a3e3764d3: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:28.284794 29307 rebalancer_tool.cc:1074] tablet b466a750cc6b4382947282d8aa6aeb1f: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:28.466348 29307 rebalancer_tool.cc:1074] tablet af693f5816e046f592fbd9b2e7ef0ad9: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:28.658601 29307 rebalancer_tool.cc:1074] tablet 843a26fc48844bfd82911b6a27833023: '297de823852d4a8da44791ab03131cd3' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:28.930876 29307 rebalancer_tool.cc:1074] tablet b5f81e46ab1f48fc8a88495b42f83640: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:29.120656 29307 rebalancer_tool.cc:1074] tablet fdd2311ec0b645f5a79098a32c6deba2: 'c107c43e538d48f58625c4320f0aa4b8' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:29.504352 29307 rebalancer_tool.cc:1074] tablet 84484d65e2bb4e23ac621a39434ce382: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:29.725435 29307 rebalancer_tool.cc:1084] tablet 14a7bead40fb4eb0bec6b00804b52595: '297de823852d4a8da44791ab03131cd3' -> '1f40a11333e44ea0a34b6bc7618af522' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:29.725576 29307 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:30.916620 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:30.916795 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server ebd8300729ec4226acc38b9cff2d8d3b
W20240502 20:46:30.916922 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:30.925338 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server 1bc8a142cfc7487bacc573afb0a92b74 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:30.925474 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:30.925549 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server 1af1e87d6ff2448d9502de9d7c2b8ef3 to server d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:46:30.925583 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:30.925617 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:30.925648 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:30.925679 29307 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:31.161171 29307 rebalancer_tool.cc:1074] tablet a547561343304137a044ce11517f6007: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '' move scheduled
I20240502 20:46:31.251327 29307 rebalancer_tool.cc:1074] tablet 6a8ce1df86914b2caf2d083d97eda4e6: 'd1f3f8e444364d6faa42d651b5dbb997' -> '' move scheduled
I20240502 20:46:31.383098 29307 rebalancer_tool.cc:1074] tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba: 'c107c43e538d48f58625c4320f0aa4b8' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:31.591873 29307 rebalancer_tool.cc:1074] tablet f00d12e425284c36ae8baed56c7cfaf6: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:31.648041 29307 rebalancer_tool.cc:1074] tablet 40df15a82a624913b3b712687d999134: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:31.835049 29307 rebalancer_tool.cc:1136] tablet 135a08f1f2ac4707bbacee81ac73a19e: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move completed: OK
I20240502 20:46:31.879180 29307 rebalancer_tool.cc:1123] tablet ff9fef57f6eb490da536170a02b96212: '297de823852d4a8da44791ab03131cd3' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move is abandoned: Illegal state: ff9fef57f6eb490da536170a02b96212: source replica 297de823852d4a8da44791ab03131cd3 does not have REPLACE attribute set
I20240502 20:46:31.890698 29307 rebalancer_tool.cc:1136] tablet d44f271f166b4c288b76689a3e3764d3: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1f40a11333e44ea0a34b6bc7618af522' move completed: OK
I20240502 20:46:31.940373 29307 rebalancer_tool.cc:1136] tablet f00c4b7954564820875a25da5834416a: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1f40a11333e44ea0a34b6bc7618af522' move completed: OK
I20240502 20:46:31.969431 29307 rebalancer_tool.cc:1136] tablet d1b9ff3c921d474fa5ad106d1c17ab00: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move completed: OK
I20240502 20:46:32.011829 29307 rebalancer_tool.cc:1136] tablet a547561343304137a044ce11517f6007: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '' move completed: OK
I20240502 20:46:32.079200 29307 rebalancer_tool.cc:1136] tablet 078a1591d9d74e31a0407aeab76911b4: '297de823852d4a8da44791ab03131cd3' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move completed: OK
I20240502 20:46:32.183502 29307 rebalancer_tool.cc:1136] tablet b466a750cc6b4382947282d8aa6aeb1f: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1bc8a142cfc7487bacc573afb0a92b74' move completed: OK
I20240502 20:46:32.255815 29307 rebalancer_tool.cc:1136] tablet fdd2311ec0b645f5a79098a32c6deba2: 'c107c43e538d48f58625c4320f0aa4b8' -> '1f40a11333e44ea0a34b6bc7618af522' move completed: Incomplete: tablet fdd2311ec0b645f5a79098a32c6deba2, TS c107c43e538d48f58625c4320f0aa4b8 -> TS 1f40a11333e44ea0a34b6bc7618af522 move failed, target replica disappeared
I20240502 20:46:32.331329 29307 rebalancer_tool.cc:1136] tablet 843a26fc48844bfd82911b6a27833023: '297de823852d4a8da44791ab03131cd3' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move completed: OK
I20240502 20:46:32.506186 29307 rebalancer_tool.cc:1136] tablet 5cf737e20ca74db1b94ccff5d60133ee: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1bc8a142cfc7487bacc573afb0a92b74' move completed: OK
I20240502 20:46:32.598776 29307 rebalancer_tool.cc:1136] tablet af693f5816e046f592fbd9b2e7ef0ad9: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move completed: OK
I20240502 20:46:32.692082 29307 rebalancer_tool.cc:1136] tablet b5f81e46ab1f48fc8a88495b42f83640: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '6706d1c0946d42a9bae62d719e1e5c31' move completed: OK
I20240502 20:46:32.734275 29307 rebalancer_tool.cc:1136] tablet 84484d65e2bb4e23ac621a39434ce382: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1bc8a142cfc7487bacc573afb0a92b74' move completed: OK
I20240502 20:46:32.755123 29307 rebalancer_tool.cc:1136] tablet 6a8ce1df86914b2caf2d083d97eda4e6: 'd1f3f8e444364d6faa42d651b5dbb997' -> '' move completed: OK
I20240502 20:46:32.768288 29307 rebalancer_tool.cc:1084] tablet a906569ab3b14520b4876bc8d5450f7d: 'c107c43e538d48f58625c4320f0aa4b8' -> '1bc8a142cfc7487bacc573afb0a92b74' move ignored: Illegal state: tablet a906569ab3b14520b4876bc8d5450f7d: replica 1bc8a142cfc7487bacc573afb0a92b74 is already present
I20240502 20:46:32.768412 29307 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): unacceptable health status UNAVAILABLE

I20240502 20:46:33.412350 29251 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } }
I20240502 20:46:33.417455 31968 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:33.418061 31968 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap complete.
I20240502 20:46:33.418354 31968 ts_tablet_manager.cc:1367] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.072s	user 0.037s	sys 0.004s
I20240502 20:46:33.419052 31968 raft_consensus.cc:348] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:46:33.419245 31968 raft_consensus.cc:729] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: LEARNER
I20240502 20:46:33.419391 31968 consensus_queue.cc:260] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:46:33.420455 31968 ts_tablet_manager.cc:1398] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:46:33.421316 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session d1f3f8e444364d6faa42d651b5dbb997-84484d65e2bb4e23ac621a39434ce382 received from {username='slave'} at 127.19.66.130:60245
I20240502 20:46:33.421428 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session d1f3f8e444364d6faa42d651b5dbb997-84484d65e2bb4e23ac621a39434ce382 on tablet 84484d65e2bb4e23ac621a39434ce382 with peer d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:46:33.424727 27227 consensus_queue.cc:237] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:46:33.430193 29251 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:33.431222 31180 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:46:33.431811 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.432384 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.434486 31914 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.435101 29251 raft_consensus.cc:2932] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.439049 26812 catalog_manager.cc:5484] T b5f81e46ab1f48fc8a88495b42f83640 P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 9 to 10, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.459013 27091 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
W20240502 20:46:33.461503 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1b9ff3c921d474fa5ad106d1c17ab00. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:33.461628 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1b9ff3c921d474fa5ad106d1c17ab00. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.462105 27359 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:33.462105 29251 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 20:46:33.462711 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.462782 30058 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:33.463160 31920 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:33.467360 31920 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.467332 29251 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.470780 27359 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.476104 26812 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P 6706d1c0946d42a9bae62d719e1e5c31 reported cstate change: config changed from index 8 to 9, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:46:33.487416 26957 consensus_queue.cc:237] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:46:33.489730 27227 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:46:33.490203 26893 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): Couldn't send request to peer ebd8300729ec4226acc38b9cff2d8d3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2dfc0bbbbca4cc28a97480007343424. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.490386 29251 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:33.490785 31736 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:33.491088 32046 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:33.494693 31736 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:46:33.495298 27227 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:46:33.498988 26809 catalog_manager.cc:5484] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 6 to 7, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:33.502897 29251 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:46:33.508898 29092 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:33.517329 29230 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 20:46:33.525532 27207 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:33.527446 27342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 20:46:33.529534 27071 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:33.546315 26957 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:33.548471 27091 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:33.549150 31736 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:46:33.551065 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.552098 29251 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:46:33.553223 31736 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:46:33.554070 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:33.556131 31736 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.557905 29251 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.562698 26809 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:33.574293 27091 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.590363 27227 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:33.592967 29117 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:33.592940 27091 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:33.593461 31180 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:33.593683 31913 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:33.594681 26957 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 20:46:33.596482 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.596781 31180 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:33.599653 30248 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.600180 26957 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.600180 29117 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.603984 26809 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.610574 31913 raft_consensus.cc:1050] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 to VOTER
I20240502 20:46:33.611265 31913 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:33.611752 27091 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.613263 29117 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:33.613831 31180 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:33.615057 27090 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:46:33.615794 26957 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 20:46:33.616899 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.617053 31914 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:46:33.618520 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:33.620860 31914 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:46:33.620880 31180 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.622169 29117 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.624056 26957 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.624814 27090 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.626363 26801 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P 1bc8a142cfc7487bacc573afb0a92b74 reported cstate change: config changed from index 8 to 9, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:33.627259 29303 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskF51BWD/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 20:46:25.677300 29308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:25.677726 29308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:25.684365 29308 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:46:27.151664 29308 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 20:46:27.879419 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.879642 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.879760 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.879822 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.879943 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.881148 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.905020 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.905949 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 01e571f10ffe4d4c88601420e733e5c7: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.906278 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.906381 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.906447 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.906549 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.906641 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.910612 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.910755 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.910852 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.910917 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:27.994158 29308 rebalancer_tool.cc:1074] tablet 5704bdfe6c02437c8e9c8c3deaadb9e3: '297de823852d4a8da44791ab03131cd3' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:28.042662 29308 rebalancer_tool.cc:1074] tablet a547561343304137a044ce11517f6007: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:28.140733 29308 rebalancer_tool.cc:1074] tablet 14a7bead40fb4eb0bec6b00804b52595: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:28.233896 29308 rebalancer_tool.cc:1074] tablet 078a1591d9d74e31a0407aeab76911b4: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:28.489377 29308 rebalancer_tool.cc:1074] tablet f00d12e425284c36ae8baed56c7cfaf6: '297de823852d4a8da44791ab03131cd3' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:28.634038 29308 rebalancer_tool.cc:1074] tablet fdd2311ec0b645f5a79098a32c6deba2: 'c107c43e538d48f58625c4320f0aa4b8' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:28.785336 29308 rebalancer_tool.cc:1074] tablet bf189c19164442e5af0827f1fb7dc085: 'd1f3f8e444364d6faa42d651b5dbb997' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:28.996392 29308 rebalancer_tool.cc:1074] tablet f00c4b7954564820875a25da5834416a: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:29.155562 29308 rebalancer_tool.cc:1074] tablet 62ebc6576a4e4a108de55a4e9d7ff6b8: '297de823852d4a8da44791ab03131cd3' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:29.301733 29308 rebalancer_tool.cc:1074] tablet a2b61909194d488682afd91c0f796405: 'c107c43e538d48f58625c4320f0aa4b8' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:29.691612 29308 rebalancer_tool.cc:1074] tablet 5cf737e20ca74db1b94ccff5d60133ee: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:29.949441 29308 rebalancer_tool.cc:1084] tablet b5f81e46ab1f48fc8a88495b42f83640: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '6706d1c0946d42a9bae62d719e1e5c31' move ignored: Illegal state: tablet b5f81e46ab1f48fc8a88495b42f83640: replica 6706d1c0946d42a9bae62d719e1e5c31 is already present
I20240502 20:46:29.949575 29308 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:31.310353 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:31.310457 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:31.310564 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:31.310679 29308 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:31.392378 29308 rebalancer_tool.cc:1074] tablet 8deaa1f751034c15acd6ccaad37536c6: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '' move scheduled
I20240502 20:46:31.489740 29308 rebalancer_tool.cc:1074] tablet 6a8ce1df86914b2caf2d083d97eda4e6: 'd1f3f8e444364d6faa42d651b5dbb997' -> '' move scheduled
I20240502 20:46:31.617569 29308 rebalancer_tool.cc:1074] tablet ff9fef57f6eb490da536170a02b96212: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:31.657861 29308 rebalancer_tool.cc:1074] tablet d1b9ff3c921d474fa5ad106d1c17ab00: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:31.704521 29308 rebalancer_tool.cc:1074] tablet a906569ab3b14520b4876bc8d5450f7d: 'c107c43e538d48f58625c4320f0aa4b8' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:31.812815 29308 rebalancer_tool.cc:1074] tablet cd1b1316da2d47938164ac5b75f11bae: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:31.885269 29308 rebalancer_tool.cc:1136] tablet fdd2311ec0b645f5a79098a32c6deba2: 'c107c43e538d48f58625c4320f0aa4b8' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move completed: OK
I20240502 20:46:31.916407 29308 rebalancer_tool.cc:1136] tablet f00d12e425284c36ae8baed56c7cfaf6: '297de823852d4a8da44791ab03131cd3' -> '1bc8a142cfc7487bacc573afb0a92b74' move completed: OK
I20240502 20:46:32.002838 29308 rebalancer_tool.cc:1136] tablet 078a1591d9d74e31a0407aeab76911b4: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1f40a11333e44ea0a34b6bc7618af522' move completed: Incomplete: tablet 078a1591d9d74e31a0407aeab76911b4, TS ebd8300729ec4226acc38b9cff2d8d3b -> TS 1f40a11333e44ea0a34b6bc7618af522 move failed, target replica disappeared
I20240502 20:46:32.141331 29308 rebalancer_tool.cc:1136] tablet 14a7bead40fb4eb0bec6b00804b52595: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move completed: OK
I20240502 20:46:32.494974 29308 rebalancer_tool.cc:1136] tablet bf189c19164442e5af0827f1fb7dc085: 'd1f3f8e444364d6faa42d651b5dbb997' -> '6706d1c0946d42a9bae62d719e1e5c31' move completed: OK
I20240502 20:46:32.513716 29308 rebalancer_tool.cc:1136] tablet f00c4b7954564820875a25da5834416a: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1bc8a142cfc7487bacc573afb0a92b74' move completed: Incomplete: tablet f00c4b7954564820875a25da5834416a, TS ebd8300729ec4226acc38b9cff2d8d3b -> TS 1bc8a142cfc7487bacc573afb0a92b74 move failed, target replica disappeared
I20240502 20:46:32.673480 29308 rebalancer_tool.cc:1136] tablet 62ebc6576a4e4a108de55a4e9d7ff6b8: '297de823852d4a8da44791ab03131cd3' -> '1f40a11333e44ea0a34b6bc7618af522' move completed: Incomplete: tablet 62ebc6576a4e4a108de55a4e9d7ff6b8, TS 297de823852d4a8da44791ab03131cd3 -> TS 1f40a11333e44ea0a34b6bc7618af522 move failed, target replica disappeared
I20240502 20:46:32.720572 29308 rebalancer_tool.cc:1136] tablet a2b61909194d488682afd91c0f796405: 'c107c43e538d48f58625c4320f0aa4b8' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move completed: OK
I20240502 20:46:32.802027 29308 rebalancer_tool.cc:1136] tablet 5704bdfe6c02437c8e9c8c3deaadb9e3: '297de823852d4a8da44791ab03131cd3' -> '1bc8a142cfc7487bacc573afb0a92b74' move completed: OK
I20240502 20:46:33.144269 29308 rebalancer_tool.cc:1136] tablet 6a8ce1df86914b2caf2d083d97eda4e6: 'd1f3f8e444364d6faa42d651b5dbb997' -> '' move completed: OK
I20240502 20:46:33.261248 29308 rebalancer_tool.cc:1084] tablet 40df15a82a624913b3b712687d999134: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move ignored: Illegal state: tablet 40df15a82a624913b3b712687d999134: replica 1af1e87d6ff2448d9502de9d7c2b8ef3 is already present
I20240502 20:46:33.261379 29308 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): unacceptable health status UNAVAILABLE

W20240502 20:46:33.640406 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:46:33.660770 31648 ts_tablet_manager.cc:902] T f00c4b7954564820875a25da5834416a P 6706d1c0946d42a9bae62d719e1e5c31: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
F20240502 20:46:33.661029 31648 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.queue_time_us, tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.queue_time_us) (tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.run_cpu_time_us, tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.run_cpu_time_us) (tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.run_wall_time_us, tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.run_wall_time_us) (tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.queue_time_us, tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.queue_time_us) (tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.run_cpu_time_us, tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.run_cpu_time_us) (tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.run_wall_time_us, tablet-download-pool-14a7bead40fb4eb0bec6b00804b52595.run_wall_time_us) (tablet-download-pool-5586b9e60b544aed87c24f7b3ce0198c.queue_time_us, tablet-download-pool-5586b9e60b544aed87c24f7b3ce0198c.queue_time_us) (tablet-download-pool-5586b9e60b544aed87c24f7b3ce0198c.run_cpu_time_us, tablet-download-pool-5586b9e60b544aed87c24f7b3ce0198c.run_cpu_time_us) (tablet-download-pool-5586b9e60b544aed87c24f7b3ce0198c.run_wall_time_us, tablet-download-pool-5586b9e60b544aed87c24f7b3ce0198c.run_wall_time_us) (tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.queue_time_us, tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.queue_time_us) (tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.run_cpu_time_us, tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.run_cpu_time_us) (tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.run_wall_time_us, tablet-download-pool-62ebc6576a4e4a108de55a4e9d7ff6b8.run_wall_time_us) (tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.queue_time_us, tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.queue_time_us) (tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_cpu_time_us, tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_cpu_time_us) (tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_wall_time_us, tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_wall_time_us) (tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.queue_time_us, tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.queue_time_us) (tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_cpu_time_us, tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_cpu_time_us) (tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_wall_time_us, tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_wall_time_us) (tablet-download-pool-a547561343304137a044ce11517f6007.queue_time_us, tablet-download-pool-a547561343304137a044ce11517f6007.queue_time_us) (tablet-download-pool-a547561343304137a044ce11517f6007.run_cpu_time_us, tablet-download-pool-a547561343304137a044ce11517f6007.run_cpu_time_us) (tablet-download-pool-a547561343304137a044ce11517f6007.run_wall_time_us, tablet-download-pool-a547561343304137a044ce11517f6007.run_wall_time_us) (tablet-download-pool-af693f5816e046f592fbd9b2e7ef0ad9.queue_time_us, tablet-download-pool-af693f5816e046f592fbd9b2e7ef0ad9.queue_time_us) (tablet-download-pool-af693f5816e046f592fbd9b2e7ef0ad9.run_cpu_time_us, tablet-download-pool-af693f5816e046f592fbd9b2e7ef0ad9.run_cpu_time_us) (tablet-download-pool-af693f5816e046f592fbd9b2e7ef0ad9.run_wall_time_us, tablet-download-pool-af693f5816e046f592fbd9b2e7ef0ad9.run_wall_time_us) (tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.queue_time_us, tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.queue_time_us) (tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.run_cpu_time_us, tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.run_cpu_time_us) (tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.run_wall_time_us, tablet-download-pool-b5f81e46ab1f48fc8a88495b42f83640.run_wall_time_us) (tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.queue_time_us, tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.queue_time_us) (tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_cpu_time_us, tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_cpu_time_us) (tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_wall_time_us, tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_wall_time_us) (tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.queue_time_us, tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.queue_time_us) (tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.run_cpu_time_us, tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.run_cpu_time_us) (tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.run_wall_time_us, tablet-download-pool-d1b9ff3c921d474fa5ad106d1c17ab00.run_wall_time_us) (tablet-download-pool-f00c4b7954564820875a25da5834416a.queue_time_us, tablet-download-pool-f00c4b7954564820875a25da5834416a.queue_time_us) (tablet-download-pool-f2dfc0bbbbca4cc28a97480007343424.queue_time_us, tablet-download-pool-f2dfc0bbbbca4cc28a97480007343424.queue_time_us) (tablet-download-pool-f2dfc0bbbbca4cc28a97480007343424.run_cpu_time_us, tablet-download-pool-f2dfc0bbbbca4cc28a97480007343424.run_cpu_time_us) (tablet-download-pool-f2dfc0bbbbca4cc28a97480007343424.run_wall_time_us, tablet-download-pool-f2dfc0bbbbca4cc28a97480007343424.run_wall_time_us) (tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.queue_time_us, tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.queue_time_us) (tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_cpu_time_us, tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_cpu_time_us) (tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_wall_time_us, tablet-download-pool-fdd2311ec0b645f5a79098a32c6deba2.run_wall_time_us) (tablet-download-pool-ff9fef57f6eb490da536170a02b96212.queue_time_us, tablet-download-pool-ff9fef57f6eb490da536170a02b96212.queue_time_us) (tablet-download-pool-ff9fef57f6eb490da536170a02b96212.run_cpu_time_us, tablet-download-pool-ff9fef57f6eb490da536170a02b96212.run_cpu_time_us) (tablet-download-pool-ff9fef57f6eb490da536170a02b96212.run_wall_time_us, tablet-download-pool-ff9fef57f6eb490da536170a02b96212.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7f093b231dcd  google::LogMessage::Fail() at ??:0
    @     0x7f093b235b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f093b2317cc  google::LogMessage::Flush() at ??:0
    @     0x7f093b232f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f093c69b044  kudu::TraceMetrics::InternName() at ??:0
    @     0x7f093c67ff98  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7f093c67defc  kudu::ThreadPoolBuilder::Build() at ??:0
W20240502 20:46:33.688407 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
    @     0x7f093f23077a  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7f093f239a45  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7f093f2ba76f  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 20:46:33.701395 27090 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "367543df62e048268908eb0d84c14876"
dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35498
I20240502 20:46:33.701650 27090 raft_consensus.cc:595] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Received request to transfer leadership
    @     0x7f093f2ce4f6  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7f093f2b9226  _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 20:46:33.709874 29251 raft_consensus.cc:1203] T 6a8ce1df86914b2caf2d083d97eda4e6 P 6706d1c0946d42a9bae62d719e1e5c31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
    @     0x7f093f2c8c0a  _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x561fbc3fc75c  std::function<>::operator()() at ??:0
    @     0x7f093c682e06  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f093c6836ef  _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
    @     0x7f093c68503b  _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x561fbc3fc75c  std::function<>::operator()() at ??:0
    @     0x7f093c672807  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f093ceb06db  start_thread at ??:0
    @     0x7f093a3e871f  clone at ??:0
I20240502 20:46:33.742405 31913 raft_consensus.cc:1050] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:33.742993 31913 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:33.745113 27359 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:46:33.745594 31180 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:46:33.747464 29117 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:46:33.748138 31180 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:46:33.754756 31914 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:33.758515 26805 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 10 to 11, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:33.758689 29117 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:33.759032 27359 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:33.763643 27090 consensus_queue.cc:237] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:33.766539 26957 raft_consensus.cc:1261] T ff9fef57f6eb490da536170a02b96212 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:46:33.767527 30058 consensus_queue.cc:1035] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:46:33.768301 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.770763 29117 raft_consensus.cc:1203] T a2b61909194d488682afd91c0f796405 P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 20:46:33.771437 31181 raft_consensus.cc:1050] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 to VOTER
I20240502 20:46:33.771822 31181 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:46:33.774031 27359 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 20:46:33.774339 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:33.774482 31181 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:33.774551 29117 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:33.776562 31181 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:33.779393 31181 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:33.782172 29117 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P 1bc8a142cfc7487bacc573afb0a92b74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:33.782754 27361 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:46:33.783942 26802 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 7 to 8, 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:33.810691 31663 ts_tablet_manager.cc:902] T 5586b9e60b544aed87c24f7b3ce0198c P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
F20240502 20:46:33.810973 31663 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.queue_time_us, tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.queue_time_us) (tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_cpu_time_us, tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_cpu_time_us) (tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_wall_time_us, tablet-download-pool-078a1591d9d74e31a0407aeab76911b4.run_wall_time_us) (tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.queue_time_us, tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.queue_time_us) (tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.run_cpu_time_us, tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.run_cpu_time_us) (tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.run_wall_time_us, tablet-download-pool-135a08f1f2ac4707bbacee81ac73a19e.run_wall_time_us) (tablet-download-pool-367543df62e048268908eb0d84c14876.queue_time_us, tablet-download-pool-367543df62e048268908eb0d84c14876.queue_time_us) (tablet-download-pool-367543df62e048268908eb0d84c14876.run_cpu_time_us, tablet-download-pool-367543df62e048268908eb0d84c14876.run_cpu_time_us) (tablet-download-pool-367543df62e048268908eb0d84c14876.run_wall_time_us, tablet-download-pool-367543df62e048268908eb0d84c14876.run_wall_time_us) (tablet-download-pool-5586b9e60b544aed87c24f7b3ce0198c.queue_time_us, tablet-download-pool-5586b9e60b544aed87c24f7b3ce0198c.queue_time_us) (tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.queue_time_us, tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.queue_time_us) (tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.run_cpu_time_us, tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.run_cpu_time_us) (tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.run_wall_time_us, tablet-download-pool-5704bdfe6c02437c8e9c8c3deaadb9e3.run_wall_time_us) (tablet-download-pool-5cf737e20ca74db1b94ccff5d60133ee.queue_time_us, tablet-download-pool-5cf737e20ca74db1b94ccff5d60133ee.queue_time_us) (tablet-download-pool-5cf737e20ca74db1b94ccff5d60133ee.run_cpu_time_us, tablet-download-pool-5cf737e20ca74db1b94ccff5d60133ee.run_cpu_time_us) (tablet-download-pool-5cf737e20ca74db1b94ccff5d60133ee.run_wall_time_us, tablet-download-pool-5cf737e20ca74db1b94ccff5d60133ee.run_wall_time_us) (tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.queue_time_us, tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.queue_time_us) (tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_cpu_time_us, tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_cpu_time_us) (tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_wall_time_us, tablet-download-pool-6a8ce1df86914b2caf2d083d97eda4e6.run_wall_time_us) (tablet-download-pool-84484d65e2bb4e23ac621a39434ce382.queue_time_us, tablet-download-pool-84484d65e2bb4e23ac621a39434ce382.queue_time_us) (tablet-download-pool-84484d65e2bb4e23ac621a39434ce382.run_cpu_time_us, tablet-download-pool-84484d65e2bb4e23ac621a39434ce382.run_cpu_time_us) (tablet-download-pool-84484d65e2bb4e23ac621a39434ce382.run_wall_time_us, tablet-download-pool-84484d65e2bb4e23ac621a39434ce382.run_wall_time_us) (tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.queue_time_us, tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.queue_time_us) (tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_cpu_time_us, tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_cpu_time_us) (tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_wall_time_us, tablet-download-pool-8deaa1f751034c15acd6ccaad37536c6.run_wall_time_us) (tablet-download-pool-a2b61909194d488682afd91c0f796405.queue_time_us, tablet-download-pool-a2b61909194d488682afd91c0f796405.queue_time_us) (tablet-download-pool-a2b61909194d488682afd91c0f796405.run_cpu_time_us, tablet-download-pool-a2b61909194d488682afd91c0f796405.run_cpu_time_us) (tablet-download-pool-a2b61909194d488682afd91c0f796405.run_wall_time_us, tablet-download-pool-a2b61909194d488682afd91c0f796405.run_wall_time_us) (tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.queue_time_us, tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.queue_time_us) (tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.run_cpu_time_us, tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.run_cpu_time_us) (tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.run_wall_time_us, tablet-download-pool-a906569ab3b14520b4876bc8d5450f7d.run_wall_time_us) (tablet-download-pool-b466a750cc6b4382947282d8aa6aeb1f.queue_time_us, tablet-download-pool-b466a750cc6b4382947282d8aa6aeb1f.queue_time_us) (tablet-download-pool-b466a750cc6b4382947282d8aa6aeb1f.run_cpu_time_us, tablet-download-pool-b466a750cc6b4382947282d8aa6aeb1f.run_cpu_time_us) (tablet-download-pool-b466a750cc6b4382947282d8aa6aeb1f.run_wall_time_us, tablet-download-pool-b466a750cc6b4382947282d8aa6aeb1f.run_wall_time_us) (tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.queue_time_us, tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.queue_time_us) (tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_cpu_time_us, tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_cpu_time_us) (tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_wall_time_us, tablet-download-pool-bf189c19164442e5af0827f1fb7dc085.run_wall_time_us) (tablet-download-pool-f00c4b7954564820875a25da5834416a.queue_time_us, tablet-download-pool-f00c4b7954564820875a25da5834416a.queue_time_us) (tablet-download-pool-f00c4b7954564820875a25da5834416a.run_cpu_time_us, tablet-download-pool-f00c4b7954564820875a25da5834416a.run_cpu_time_us) (tablet-download-pool-f00c4b7954564820875a25da5834416a.run_wall_time_us, tablet-download-pool-f00c4b7954564820875a25da5834416a.run_wall_time_us) (tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.queue_time_us, tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.queue_time_us) (tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.run_cpu_time_us, tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.run_cpu_time_us) (tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.run_wall_time_us, tablet-download-pool-f00d12e425284c36ae8baed56c7cfaf6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7f4d3ae09dcd  google::LogMessage::Fail() at ??:0
I20240502 20:46:33.821422 27361 raft_consensus.cc:1261] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Refusing update from remote peer 1bc8a142cfc7487bacc573afb0a92b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 20:46:33.822486 31992 consensus_queue.cc:1035] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 20:46:33.829447 29048 consensus_peers.cc:485] T 8deaa1f751034c15acd6ccaad37536c6 P 1bc8a142cfc7487bacc573afb0a92b74 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
    @     0x7f4d3ae0db93  google::LogMessage::SendToLog() at ??:0
    @     0x7f4d3ae097cc  google::LogMessage::Flush() at ??:0
    @     0x7f4d3ae0af59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f4d3c273044  kudu::TraceMetrics::InternName() at ??:0
    @     0x7f4d3c257f98  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7f4d3c255efc  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7f4d3ee0877a  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 20:46:33.875036 27361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595"
dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38740
I20240502 20:46:33.875257 27361 raft_consensus.cc:595] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Received request to transfer leadership
    @     0x7f4d3ee11a45  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7f4d3ee9276f  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7f4d3eea64f6  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7f4d3ee91226  _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 20:46:33.909296 32258 ts_tablet_manager.cc:902] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:46:33.909972 32258 tablet_copy_client.cc:308] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
    @     0x7f4d3eea0c0a  _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x55c30078975c  std::function<>::operator()() at ??:0
I20240502 20:46:33.913197 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet b5f81e46ab1f48fc8a88495b42f83640 from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:45531)
I20240502 20:46:33.913331 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet b5f81e46ab1f48fc8a88495b42f83640 from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:45531: session id = 297de823852d4a8da44791ab03131cd3-b5f81e46ab1f48fc8a88495b42f83640
I20240502 20:46:33.913362 32261 ts_tablet_manager.cc:902] T d1b9ff3c921d474fa5ad106d1c17ab00 P 1bc8a142cfc7487bacc573afb0a92b74: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:33.914384 27249 tablet_copy_source_session.cc:205] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:33.920064 32258 ts_tablet_manager.cc:1875] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:46:33.923621 32258 ts_tablet_manager.cc:1888] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 20:46:33.923836 32258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5f81e46ab1f48fc8a88495b42f83640. 1 dirs total, 0 dirs full, 0 dirs failed
    @     0x7f4d3c25ae06  kudu::ThreadPool::DispatchThread() at ??:0
I20240502 20:46:33.929791 32258 tablet_copy_client.cc:783] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:33.929996 32258 tablet_copy_client.cc:647] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:33.938278 32258 tablet_copy_client.cc:523] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:33.940850 32258 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
    @     0x7f4d3c25b6ef  _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240502 20:46:33.948547 27090 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "367543df62e048268908eb0d84c14876"
dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35538
I20240502 20:46:33.948752 27090 raft_consensus.cc:595] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:33.950416 31921 raft_consensus.cc:979] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997: : Instructing follower ebd8300729ec4226acc38b9cff2d8d3b to start an election
I20240502 20:46:33.950542 31921 raft_consensus.cc:1067] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Signalling peer ebd8300729ec4226acc38b9cff2d8d3b to start an election
I20240502 20:46:33.952309 32290 ts_tablet_manager.cc:902] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:46:33.952292 27361 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "367543df62e048268908eb0d84c14876"
dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
 from {username='slave'} at 127.19.66.130:43999
I20240502 20:46:33.952548 27361 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:33.952641 27361 raft_consensus.cc:3037] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Advancing to term 2
    @     0x7f4d3c25d03b  _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240502 20:46:33.953222 32290 tablet_copy_client.cc:308] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:46:33.953928 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet d1b9ff3c921d474fa5ad106d1c17ab00 from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:37781)
I20240502 20:46:33.954067 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet d1b9ff3c921d474fa5ad106d1c17ab00 from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:37781: session id = 297de823852d4a8da44791ab03131cd3-d1b9ff3c921d474fa5ad106d1c17ab00
    @     0x55c30078975c  std::function<>::operator()() at ??:0
I20240502 20:46:33.954733 27361 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:33.955621 27112 tablet_copy_source_session.cc:205] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:33.956935 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997"
I20240502 20:46:33.957329 27090 raft_consensus.cc:3032] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:33.957402 27090 raft_consensus.cc:729] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:33.957468 32290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b9ff3c921d474fa5ad106d1c17ab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:46:33.957676 27090 consensus_queue.cc:260] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:33.958034 27090 raft_consensus.cc:3037] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:33.959791 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
I20240502 20:46:33.961764 32290 tablet_copy_client.cc:783] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:33.961988 32290 tablet_copy_client.cc:647] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
    @     0x7f4d3c24a807  kudu::Thread::SuperviseThread() at ??:0
I20240502 20:46:33.963661 32290 tablet_copy_client.cc:523] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
    @     0x7f4d3ca886db  start_thread at ??:0
I20240502 20:46:33.965953 32290 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
W20240502 20:46:33.965934 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:46:33.966604 32284 ts_tablet_manager.cc:902] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b: Initiating tablet copy from peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:33.967694 27361 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 election: Requested vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:33.970115 32284 tablet_copy_client.cc:308] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:38835
W20240502 20:46:33.970153 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:33.970244 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:33.970682 31211 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 20:46:33.971065 26980 tablet_copy_service.cc:140] P 297de823852d4a8da44791ab03131cd3: Received BeginTabletCopySession request for tablet f2dfc0bbbbca4cc28a97480007343424 from peer ebd8300729ec4226acc38b9cff2d8d3b ({username='slave'} at 127.19.66.132:51721)
I20240502 20:46:33.971208 26980 tablet_copy_service.cc:161] P 297de823852d4a8da44791ab03131cd3: Beginning new tablet copy session on tablet f2dfc0bbbbca4cc28a97480007343424 from peer ebd8300729ec4226acc38b9cff2d8d3b at {username='slave'} at 127.19.66.132:51721: session id = ebd8300729ec4226acc38b9cff2d8d3b-f2dfc0bbbbca4cc28a97480007343424
I20240502 20:46:33.972213 26980 tablet_copy_source_session.cc:205] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 20:46:33.978152 32284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2dfc0bbbbca4cc28a97480007343424. 1 dirs total, 0 dirs full, 0 dirs failed
    @     0x7f4d39fc071f  clone at ??:0
I20240502 20:46:33.989328 32284 tablet_copy_client.cc:783] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Starting download of 0 data blocks...
I20240502 20:46:33.989706 32284 tablet_copy_client.cc:647] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Starting download of 1 WAL segments...
I20240502 20:46:33.992475 32284 tablet_copy_client.cc:523] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:46:33.998054 32284 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:46:34.008749 32258 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:34.022573 32258 tablet_bootstrap.cc:492] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:46:34.022843 32258 ts_tablet_manager.cc:1367] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.082s	user 0.032s	sys 0.001s
I20240502 20:46:34.023464 32258 raft_consensus.cc:348] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:46:34.023617 32258 raft_consensus.cc:729] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:46:34.023742 32258 consensus_queue.cc:260] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:46:34.024070 32258 ts_tablet_manager.cc:1398] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:34.025559 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-b5f81e46ab1f48fc8a88495b42f83640 received from {username='slave'} at 127.19.66.129:45531
I20240502 20:46:34.025648 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 297de823852d4a8da44791ab03131cd3-b5f81e46ab1f48fc8a88495b42f83640 on tablet b5f81e46ab1f48fc8a88495b42f83640 with peer 297de823852d4a8da44791ab03131cd3
I20240502 20:46:34.074637 27361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595"
dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38772
I20240502 20:46:34.074842 27361 raft_consensus.cc:595] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Received request to transfer leadership
I20240502 20:46:34.075057 32290 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:34.075430 32290 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:46:34.075706 32290 ts_tablet_manager.cc:1367] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.110s	user 0.034s	sys 0.003s
I20240502 20:46:34.076282 32290 raft_consensus.cc:348] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:46:34.076459 32290 raft_consensus.cc:729] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:46:34.076593 32290 consensus_queue.cc:260] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:46:34.076958 32290 ts_tablet_manager.cc:1398] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:46:34.078158 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-d1b9ff3c921d474fa5ad106d1c17ab00 received from {username='slave'} at 127.19.66.129:37781
I20240502 20:46:34.078251 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 297de823852d4a8da44791ab03131cd3-d1b9ff3c921d474fa5ad106d1c17ab00 on tablet d1b9ff3c921d474fa5ad106d1c17ab00 with peer 297de823852d4a8da44791ab03131cd3
I20240502 20:46:34.082728 27227 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
W20240502 20:46:34.085402 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:34.086558 27090 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:34.087051 31180 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:46:34.089218 31180 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:34.091516 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84484d65e2bb4e23ac621a39434ce382 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:46:34.092893 26809 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 9 to 10, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:46:34.093493 27091 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } }
I20240502 20:46:34.100941 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet 84484d65e2bb4e23ac621a39434ce382 with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38912
I20240502 20:46:34.102959 32284 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:46:34.103382 32284 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap complete.
I20240502 20:46:34.103617 32284 ts_tablet_manager.cc:1367] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.106s	user 0.038s	sys 0.000s
I20240502 20:46:34.104562 32284 raft_consensus.cc:348] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:46:34.104753 32284 raft_consensus.cc:729] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: LEARNER
I20240502 20:46:34.104866 32284 consensus_queue.cc:260] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:46:34.105242 32284 ts_tablet_manager.cc:1398] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 20:46:34.105597 32356 tablet_replica.cc:330] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:46:34.105759 32356 raft_consensus.cc:2229] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:46:34.106071 32356 raft_consensus.cc:2258] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:46:34.107228 32356 ts_tablet_manager.cc:1875] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:46:34.110935 32356 ts_tablet_manager.cc:1888] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 20:46:34.111042 32356 log.cc:1196] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/84484d65e2bb4e23ac621a39434ce382
I20240502 20:46:34.112902 26980 tablet_copy_service.cc:342] P 297de823852d4a8da44791ab03131cd3: Request end of tablet copy session ebd8300729ec4226acc38b9cff2d8d3b-f2dfc0bbbbca4cc28a97480007343424 received from {username='slave'} at 127.19.66.132:51721
I20240502 20:46:34.112999 26980 tablet_copy_service.cc:433] P 297de823852d4a8da44791ab03131cd3: ending tablet copy session ebd8300729ec4226acc38b9cff2d8d3b-f2dfc0bbbbca4cc28a97480007343424 on tablet f2dfc0bbbbca4cc28a97480007343424 with peer ebd8300729ec4226acc38b9cff2d8d3b
I20240502 20:46:34.113365 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet 84484d65e2bb4e23ac621a39434ce382 (table rebalance_test_table_4 [id=46678b98efdf4c8b96d211c154840044]) successfully deleted
W20240502 20:46:34.118182 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:34.140856 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:34.167850 27159 connection.cc:535] client connection to 127.19.66.136:41529 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:34.167915 26892 connection.cc:535] client connection to 127.19.66.136:41529 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:34.168040 27296 connection.cc:535] client connection to 127.19.66.136:41529 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:34.171725 31934 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:46:34.168895 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.136:41529 (local address 127.19.66.131:42923)
0502 20:46:34.169159 (+   264us) negotiation.cc:107] Waiting for socket to connect
0502 20:46:34.169177 (+    18us) client_negotiation.cc:174] Beginning negotiation
0502 20:46:34.169308 (+   131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:46:34.170965 (+  1657us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":150}
W20240502 20:46:34.171756 27025 connection.cc:535] client connection to 127.19.66.136:41529 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:34.171953 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.172022 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.172959 31923 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:46:34.169222 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.136:41529 (local address 127.19.66.132:45641)
0502 20:46:34.169488 (+   266us) negotiation.cc:107] Waiting for socket to connect
0502 20:46:34.169506 (+    18us) client_negotiation.cc:174] Beginning negotiation
0502 20:46:34.169626 (+   120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:46:34.172344 (+  2718us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":160}
W20240502 20:46:34.173220 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.173310 27296 consensus_peers.cc:485] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.173895 31737 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:46:34.169443 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.136:41529 (local address 127.19.66.129:44173)
0502 20:46:34.169817 (+   374us) negotiation.cc:107] Waiting for socket to connect
0502 20:46:34.169835 (+    18us) client_negotiation.cc:174] Beginning negotiation
0502 20:46:34.169922 (+    87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:46:34.173326 (+  3404us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":275}
W20240502 20:46:34.175081 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.175971 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.176056 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:34.176095 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.176148 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.176188 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.176330 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:34.197867 31211 raft_consensus.cc:979] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b: : Instructing follower 297de823852d4a8da44791ab03131cd3 to start an election
I20240502 20:46:34.198071 31211 raft_consensus.cc:1067] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Signalling peer 297de823852d4a8da44791ab03131cd3 to start an election
I20240502 20:46:34.199298 26960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595"
dest_uuid: "297de823852d4a8da44791ab03131cd3"
 from {username='slave'} at 127.19.66.132:51721
I20240502 20:46:34.199596 26960 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 20:46:34.199682 26960 raft_consensus.cc:3037] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:34.201225 26960 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:34.203281 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b"
I20240502 20:46:34.203492 27361 raft_consensus.cc:3032] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 20:46:34.203560 27361 raft_consensus.cc:729] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Running, Role: LEADER
I20240502 20:46:34.203771 27361 consensus_queue.cc:260] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:34.204023 27361 raft_consensus.cc:3037] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Advancing to term 2
I20240502 20:46:34.205451 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:46:34.205657 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:34.205859 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:34.207960 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:34.208117 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:34.208209 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:34.208444 32047 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 20:46:34.208778 26960 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 2 election: Requested vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
W20240502 20:46:34.258597 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.307315 27297 connection.cc:535] client connection to 127.19.66.135:46845 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:34.307844 27160 connection.cc:535] client connection to 127.19.66.135:46845 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:34.307842 27026 connection.cc:535] client connection to 127.19.66.135:46845 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 20:46:34.308949 31923 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:46:34.308217 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.135:46845 (local address 127.19.66.132:41871)
0502 20:46:34.308775 (+   558us) negotiation.cc:107] Waiting for socket to connect
0502 20:46:34.308835 (+    60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":457}
W20240502 20:46:34.309105 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.309130 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection reset by peer (error 104)
W20240502 20:46:34.309170 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.309206 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.309198 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.309209 27026 consensus_peers.cc:462] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:34.309242 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.309271 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.309274 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.323671 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.326534 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:34.381440 26957 raft_consensus.cc:1203] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 20:46:34.385883 26957 raft_consensus.cc:1203] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 20:46:34.389496 27361 raft_consensus.cc:1203] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 20:46:34.395143 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:34.429013 26936 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 20:46:34.433629 27207 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 20:46:34.436442 27342 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 20:46:34.436777 27071 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 20:46:34.475374 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.491719 27026 consensus_peers.cc:462] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:46:34.502063 29305 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskF51BWD/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 20:46:25.632390 29309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 20:46:25.632730 29309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 20:46:25.637639 29309 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 20:46:27.151127 29309 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 20:46:27.936219 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.936480 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.936566 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.936619 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.936738 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.936789 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.936844 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.936898 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 01e571f10ffe4d4c88601420e733e5c7: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.936975 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.937036 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.937122 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.937176 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:27.937233 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.937292 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.937343 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:27.937399 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 01e571f10ffe4d4c88601420e733e5c7: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:27.937458 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:27.937515 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:27.985302 29309 rebalancer_tool.cc:1084] tablet 367543df62e048268908eb0d84c14876: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } }
I20240502 20:46:27.985522 29309 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:28.766973 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:28.767313 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:28.767405 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:28.767438 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:28.767514 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:28.767565 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:28.767645 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:28.767699 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:28.767753 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:28.767812 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b230b3b28de45dcade302c6a436ebad: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:28.767858 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:28.767949 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:46:28.767998 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:28.768039 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fde883c6e2f44181aad0a49a138415ab: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:28.943486 29309 rebalancer_tool.cc:1074] tablet 078a1591d9d74e31a0407aeab76911b4: '297de823852d4a8da44791ab03131cd3' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:28.997309 29309 rebalancer_tool.cc:1074] tablet 367543df62e048268908eb0d84c14876: 'd1f3f8e444364d6faa42d651b5dbb997' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:29.125873 29309 rebalancer_tool.cc:1074] tablet 5586b9e60b544aed87c24f7b3ce0198c: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:29.304095 29309 rebalancer_tool.cc:1074] tablet 8deaa1f751034c15acd6ccaad37536c6: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:29.594796 29309 rebalancer_tool.cc:1074] tablet b5f81e46ab1f48fc8a88495b42f83640: '297de823852d4a8da44791ab03131cd3' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:30.003485 29309 rebalancer_tool.cc:1074] tablet f2dfc0bbbbca4cc28a97480007343424: 'd1f3f8e444364d6faa42d651b5dbb997' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:30.362560 29309 rebalancer_tool.cc:1084] tablet fdd2311ec0b645f5a79098a32c6deba2: 'c107c43e538d48f58625c4320f0aa4b8' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move ignored: Illegal state: tablet fdd2311ec0b645f5a79098a32c6deba2: replica 1af1e87d6ff2448d9502de9d7c2b8ef3 is already present
I20240502 20:46:30.362668 29309 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:31.217495 31109 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.19.66.190:43237, user_credentials={real_user=slave}} blocked reactor thread for 47217.1us
W20240502 20:46:31.310920 31109 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.19.66.190:43237, user_credentials={real_user=slave}} blocked reactor thread for 47210.9us
W20240502 20:46:31.662070 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:31.662182 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server ebd8300729ec4226acc38b9cff2d8d3b
W20240502 20:46:31.662240 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:31.662307 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:31.662487 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server 297de823852d4a8da44791ab03131cd3 to server 6706d1c0946d42a9bae62d719e1e5c31
W20240502 20:46:31.662546 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table aab016c806224a07b904d99b93e2e5b9: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:31.662585 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2129159ea2e44a4ea9913945352f2626: could not find any suitable replica to move from server ebd8300729ec4226acc38b9cff2d8d3b to server 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:31.743515 29309 rebalancer_tool.cc:1074] tablet cd1b1316da2d47938164ac5b75f11bae: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1af1e87d6ff2448d9502de9d7c2b8ef3' move scheduled
I20240502 20:46:31.768070 29309 rebalancer_tool.cc:1084] tablet a906569ab3b14520b4876bc8d5450f7d: 'c107c43e538d48f58625c4320f0aa4b8' -> '1bc8a142cfc7487bacc573afb0a92b74' move ignored: Illegal state: tablet a906569ab3b14520b4876bc8d5450f7d: replica 1bc8a142cfc7487bacc573afb0a92b74 is already present
I20240502 20:46:31.768186 29309 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 20:46:32.381800 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 46678b98efdf4c8b96d211c154840044: could not find any suitable replica to move from server 1af1e87d6ff2448d9502de9d7c2b8ef3 to server 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:46:32.381938 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d9dbe3e88393443a8397e83015b69044: could not find any suitable replica to move from server c107c43e538d48f58625c4320f0aa4b8 to server ebd8300729ec4226acc38b9cff2d8d3b
W20240502 20:46:32.381983 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f67895aa93e54fa6a76a86a676d1a1ff: could not find any suitable replica to move from server 1af1e87d6ff2448d9502de9d7c2b8ef3 to server ebd8300729ec4226acc38b9cff2d8d3b
W20240502 20:46:32.382081 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc14910d53d74d46a1a9e2001d10f204: could not find any suitable replica to move from server d1f3f8e444364d6faa42d651b5dbb997 to server 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:46:32.382155 29309 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2bfbb94d5cdf44cb82f42848d797bd88: could not find any suitable replica to move from server 1f40a11333e44ea0a34b6bc7618af522 to server 297de823852d4a8da44791ab03131cd3
I20240502 20:46:32.462575 29309 rebalancer_tool.cc:1074] tablet 5cf737e20ca74db1b94ccff5d60133ee: 'c107c43e538d48f58625c4320f0aa4b8' -> '1f40a11333e44ea0a34b6bc7618af522' move scheduled
I20240502 20:46:32.587486 29309 rebalancer_tool.cc:1074] tablet a2b61909194d488682afd91c0f796405: '297de823852d4a8da44791ab03131cd3' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:32.632422 29309 rebalancer_tool.cc:1074] tablet 6a8ce1df86914b2caf2d083d97eda4e6: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '6706d1c0946d42a9bae62d719e1e5c31' move scheduled
I20240502 20:46:33.158262 29309 rebalancer_tool.cc:1074] tablet 84484d65e2bb4e23ac621a39434ce382: '1bc8a142cfc7487bacc573afb0a92b74' -> 'd1f3f8e444364d6faa42d651b5dbb997' move scheduled
I20240502 20:46:33.319200 29309 rebalancer_tool.cc:1074] tablet d1b9ff3c921d474fa5ad106d1c17ab00: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1bc8a142cfc7487bacc573afb0a92b74' move scheduled
I20240502 20:46:33.590994 29309 rebalancer_tool.cc:1136] tablet 5586b9e60b544aed87c24f7b3ce0198c: 'c107c43e538d48f58625c4320f0aa4b8' -> '6706d1c0946d42a9bae62d719e1e5c31' move completed: OK
I20240502 20:46:33.648247 29309 rebalancer_tool.cc:1123] tablet b5f81e46ab1f48fc8a88495b42f83640: '297de823852d4a8da44791ab03131cd3' -> '1f40a11333e44ea0a34b6bc7618af522' move is abandoned: Illegal state: b5f81e46ab1f48fc8a88495b42f83640: source replica 297de823852d4a8da44791ab03131cd3 does not have REPLACE attribute set
I20240502 20:46:33.666862 29309 rebalancer_tool.cc:1136] tablet 078a1591d9d74e31a0407aeab76911b4: '297de823852d4a8da44791ab03131cd3' -> '1bc8a142cfc7487bacc573afb0a92b74' move completed: OK
I20240502 20:46:33.719907 29309 rebalancer_tool.cc:1136] tablet f2dfc0bbbbca4cc28a97480007343424: 'd1f3f8e444364d6faa42d651b5dbb997' -> '6706d1c0946d42a9bae62d719e1e5c31' move completed: OK
W20240502 20:46:34.309782 32425 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 20:46:34.245666 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.135:46845 (local address 127.0.0.1:54710)
0502 20:46:34.245944 (+   278us) negotiation.cc:107] Waiting for socket to connect
0502 20:46:34.245959 (+    15us) client_negotiation.cc:174] Beginning negotiation
0502 20:46:34.246063 (+   104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 20:46:34.308830 (+ 62767us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":120,"threads_started":1}
I20240502 20:46:34.310237 29309 rebalancer_tool.cc:1123] tablet 8deaa1f751034c15acd6ccaad37536c6: 'ebd8300729ec4226acc38b9cff2d8d3b' -> '1bc8a142cfc7487bacc573afb0a92b74' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 20:46:34.317742 29309 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): unacceptable health status UNAVAILABLE

W20240502 20:46:34.534942 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:34.611932 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:34.667114 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:34.730005 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:34.747651 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:34.773319 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:46:34.804612 30058 raft_consensus.cc:1050] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:46:34.805362 30058 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:34.807838 27361 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:34.808176 26957 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:46:34.808391 31201 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:46:34.808583 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:34.809237 31201 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:46:34.809945 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.811147 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:34.811714 31201 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:46:34.812484 26957 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:46:34.814522 27361 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:46:34.816416 26807 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:46:34.830871 31914 raft_consensus.cc:1050] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:46:34.831372 31914 consensus_queue.cc:237] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:34.833662 26957 raft_consensus.cc:1261] T b5f81e46ab1f48fc8a88495b42f83640 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 20:46:34.834192 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:34.834406 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:34.834442 32576 consensus_queue.cc:1035] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:46:34.867237 31211 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:46:34.867595 31211 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:34.868448 31211 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:34.868875 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:34.869350 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:46:34.869648 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:34.869819 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:46:34.870026 31211 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:34.870043 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:34.968109 32474 raft_consensus.cc:1050] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: attempting to promote NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b to VOTER
I20240502 20:46:34.969125 32474 consensus_queue.cc:237] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } }
W20240502 20:46:34.969578 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:34.971066 27361 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:46:34.971067 27227 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 20:46:34.971575 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:46:34.971585 32047 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:34.971807 32581 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:46:34.974197 32581 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:46:34.974856 27227 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:46:34.976667 27361 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:46:34.977738 26806 catalog_manager.cc:5484] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 7 to 8, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:46:35.047905 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:35.064684 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.072418 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:35.081362 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:46:35.137349 32576 raft_consensus.cc:482] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6706d1c0946d42a9bae62d719e1e5c31)
I20240502 20:46:35.137557 32576 raft_consensus.cc:504] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:35.138147 32576 leader_election.cc:288] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:35.138828 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a547561343304137a044ce11517f6007" candidate_uuid: "c107c43e538d48f58625c4320f0aa4b8" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:46:35.139614 27159 leader_election.cc:334] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:35.139693 27159 leader_election.cc:302] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8; no voters: 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:46:35.139845 32576 raft_consensus.cc:2726] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:46:35.195259 31922 raft_consensus.cc:482] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6706d1c0946d42a9bae62d719e1e5c31)
I20240502 20:46:35.195470 31922 raft_consensus.cc:504] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
W20240502 20:46:35.195647 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:35.196061 31922 leader_election.cc:288] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:35.196627 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a547561343304137a044ce11517f6007" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8" is_pre_election: true
I20240502 20:46:35.196866 27227 raft_consensus.cc:2454] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:46:35.197338 27025 leader_election.cc:334] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:35.197477 27027 leader_election.cc:302] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:35.197679 31922 raft_consensus.cc:2781] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 20:46:35.197752 31922 raft_consensus.cc:482] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6706d1c0946d42a9bae62d719e1e5c31)
I20240502 20:46:35.197808 31922 raft_consensus.cc:3037] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:35.199256 31922 raft_consensus.cc:504] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:35.199765 31922 leader_election.cc:288] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 election: Requested vote from peers c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:35.200397 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a547561343304137a044ce11517f6007" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "c107c43e538d48f58625c4320f0aa4b8"
I20240502 20:46:35.200573 27227 raft_consensus.cc:3037] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 2 FOLLOWER]: Advancing to term 3
I20240502 20:46:35.202059 27227 raft_consensus.cc:2454] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 3.
W20240502 20:46:35.202103 27025 leader_election.cc:334] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:35.202435 27027 leader_election.cc:302] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c107c43e538d48f58625c4320f0aa4b8, d1f3f8e444364d6faa42d651b5dbb997; no voters: 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:35.202680 31922 raft_consensus.cc:2781] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 20:46:35.202847 31922 raft_consensus.cc:686] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 3 LEADER]: Becoming Leader. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Running, Role: LEADER
I20240502 20:46:35.203094 31922 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:46:35.205821 26805 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: term changed from 2 to 3, leader changed from 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) to d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130). New cstate: current_term: 3 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:46:35.227694 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:35.276098 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.327998 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:35.333669 31211 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:46:35.333863 31211 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:35.334415 31211 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:46:35.335907 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:35.336088 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:46:35.336236 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:35.336556 31211 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:35.369282 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.404992 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.407624 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.430994 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.473678 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:35.528849 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.563171 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:35.601025 27227 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 3 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 20:46:35.601884 31922 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 20:46:35.607918 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.607869 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:35.608778 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:35.608937 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:46:35.636612 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.649255 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:35.670897 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:35.720077 32713 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:35.720309 32713 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:35.721022 32713 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:35.721799 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:35.722067 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:46:35.722760 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.722836 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:35.722910 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:35.722956 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:35.723122 32713 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:35.784811 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:35.813517 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:35.824690 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:35.994050 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.042438 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:36.089586 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:36.121176 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:36.159046 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:36.162706 32474 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:36.162919 32474 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:36.163566 32474 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:36.164000 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:36.164182 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:46:36.165817 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:36.166167 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:36.166299 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:36.166607 32474 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:36.188040 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.229203 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.264519 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.310379 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.316102 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.347077 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:36.364388 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.367168 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.430202 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:36.481276 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.491212 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:36.543435 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.547776 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:36.549937 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:36.610424 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.612303 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.656672 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.713620 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.719296 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.773977 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:36.778180 31211 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:46:36.778414 31211 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:36.779234 31211 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:36.779732 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:36.780146 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:46:36.780877 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:36.780957 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:46:36.781172 31211 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:36.781529 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:36.817050 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.838186 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.885910 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:36.957269 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.960513 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:36.973469 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:37.012269 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:37.015100 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:37.031064 31211 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:46:37.031227 31211 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:37.031678 31211 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:46:37.032821 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:37.032970 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:37.033042 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:37.033249 31211 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:37.078776 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:37.131632 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:37.213124 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:37.295022 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:37.329901 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:37.357301 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:37.399236 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:37.417439 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:37.431435 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:37.460443 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:37.509450 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:46:37.522246   450 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:37.522493   450 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:37.523133   450 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:37.523694 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:37.523908 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:46:37.524856 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:37.524982 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:37.525045 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:37.525215   450 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:37.526052 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:37.536626 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:37.546435 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:37.611763 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:37.620695 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:37.676959 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:37.749927 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:37.847431 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:37.852170 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:37.871896 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 167 similar messages]
W20240502 20:46:37.881341 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:37.920601 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:37.957299 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:37.962102 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:37.990146 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:37.999683 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:46:38.010530 32474 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:38.010783 32474 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:38.011483 32474 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
W20240502 20:46:38.012140 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240502 20:46:38.012176 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:38.012383 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:46:38.012916 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:38.013003 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:38.013121 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:38.013290 32474 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:38.031579 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.072716 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.081414 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.109231 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:46:38.156092 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 20:46:38.163261 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:46:38.168412 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:38.186647 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.201578 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:38.282486 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.303795 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.319247 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:38.343510 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.357309 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:38.510643 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:38.629948 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.715961 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.754927 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:38.756480 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.765683 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:38.790695 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:38.791527 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:38.800515 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.840837 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.865867 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.868552 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.885526 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.892542 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.975360 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:38.984426 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:39.012332 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:46:39.027513 31211 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:39.027719 31211 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:39.028292 31211 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:39.029033 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:39.029340 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:46:39.029809 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:39.030409 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:39.030493 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:39.030715 31211 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:39.073124 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:39.087503 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:39.117537 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:39.137459 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:39.141765 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:39.188365 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:39.188685 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:39.252063 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:39.259589 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:39.278324 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:46:39.332492 31211 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:46:39.332666 31211 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:39.333312 31211 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:46:39.335096 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:39.335234 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:39.335331 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:39.335539 31211 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:39.386289 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:39.458856 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:39.521761 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:39.567934 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:39.580150 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:39.619861 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:39.679839 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:46:39.690687 31211 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:39.690877 31211 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:39.691426 31211 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:39.692013 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:39.692219 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:46:39.692782 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:39.693092 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:39.693166 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:39.693372 31211 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:39.741781 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:39.772117 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:46:39.791898 31211 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:46:39.792090 31211 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:39.792750 31211 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:39.793709 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:39.793732 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:46:39.794931 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:39.795032 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:46:39.795114 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:39.795271 31211 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:39.862437 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:39.863046 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:39.987051 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:39.995205 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:39.996138 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:40.003507 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:40.024253 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.029160 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:40.029160 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:40.051528 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.097673 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:40.118602 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:40.121472 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.156015 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:40.213351 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.256314 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.281575 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.323207 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:40.325191 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:40.423354 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.460052 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.472513 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:40.477128 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:40.498435 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:40.587148 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:40.610435 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:40.644251 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.684799 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.716035 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.716317 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:40.734969 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:46:40.761870 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:40.808820 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:40.933825 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:40.966740 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:41.023924 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:46:41.029021  1108 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:41.029251  1108 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:41.029915  1108 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:41.030288 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:41.030481 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:46:41.031618 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:41.031751 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:41.031816 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:41.032094  1108 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:41.044649 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:41.121217 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.172685 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:46:41.176839  1108 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:46:41.177021  1108 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:41.177603  1108 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:46:41.178831 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:41.179308 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:46:41.179383 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:41.179586  1108 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:41.185667 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.186760 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.207464 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.246178 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.253051 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:41.369421 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.423087 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:46:41.445744  1108 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:41.445923  1108 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:41.446504  1108 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:41.446955 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:41.447132 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:46:41.448299 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:41.448547 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:41.448621 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:41.448779  1108 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:41.463025 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:41.483309 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:41.483404 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.485343 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.542539 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:41.585702 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.588352 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.616044 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.637140 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.648265 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.651166 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.652065 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.721613 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:41.766701 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:41.794934 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.837217 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.868590 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:41.949016 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.017390 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:42.020735 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.035115 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:42.063582 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:42.079895 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.130127 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.168995 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:42.209194 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:42.336792 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:42.351909 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.488813 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:42.489568 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:42.516656 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.559159 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:42.560128 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.572510 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.580221 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:42.583887 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.586253 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:42.653973 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.661468 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:42.779992 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:42.783993  1254 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:42.784195  1254 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
W20240502 20:46:42.784772 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:42.784780  1254 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:42.785197 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:42.785383 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:46:42.786137 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:42.786259 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:42.786316 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:42.786564  1254 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:42.804826 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:42.836823 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.855772 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:42.871764 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:42.875363 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:42.893532 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:46:42.895814 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:42.916728 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:42.960366 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:43.092855 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:46:43.099424 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:43.108525 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.136245 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:46:43.147275 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.154876 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:43.167336 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 20:46:43.210723 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 20:46:43.216118 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.292192 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:43.327358 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:43.458382 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.462541 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.533255 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:43.551198 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.590421 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:43.645617 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.664881 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:43.676215  1254 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:46:43.676474  1254 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
W20240502 20:46:43.677151 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:43.677140  1254 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:46:43.678179 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:43.678272 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:46:43.678341 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:43.678525  1254 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:43.757143 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:43.769384 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.772297 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.787366 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.819226 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.832599 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.842092 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:43.845752  1578 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:46:43.845976  1578 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:43.846766  1578 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:43.847409 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:43.847656 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:46:43.848255 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:43.848331 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:46:43.848462 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:43.848554  1578 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:43.940795 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:43.975994 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:43.981834 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:43.985785 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.049026 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.052664 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.092646 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.094694 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.096800 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:44.190798 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:44.209723 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:44.254720 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.255453 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.256385 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.270097 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.305514 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.348703 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.355619 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 20:46:44.421327  1578 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:44.421566  1578 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:44.422286  1578 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:44.422956 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:44.423188 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:46:44.423765 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:44.423910 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:44.424006 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:44.424187  1578 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:44.425482 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:44.528020 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:44.560451 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:44.584805 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.656260 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.715258 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:44.731199 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.731285 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:44.781291 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:44.791877  1741 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:44.792049  1741 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:44.792719  1741 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:44.793056 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:44.793277 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:46:44.794380 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:44.794513 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:44.794570 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:44.794723  1741 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:44.827695 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.892220 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:44.960391 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:44.998538 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:45.007911 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:45.042059 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.043148 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:45.057564 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.061614 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:45.062757 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.069202 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:45.139173 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.140349 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:45.274080 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:45.302098 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:45.395522 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.429641 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.494361 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:45.543373 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:45.543704 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:46:45.568706 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:45.645004 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:45.674619 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:45.687671 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.717773 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.768754 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.808200 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:45.827008 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.983608 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:45.992249 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.024055 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:46.026978 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.041206 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:46.110711 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.155776 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.172154 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:46.187456 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.190739 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:46.201133 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.209270 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:46.222086 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.269150 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.308809 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.327419 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.347805 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.396162 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:46:46.425190  1431 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:46.425423  1431 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:46.426165  1431 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:46.426641 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:46.426827 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:46:46.427417 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:46.427547 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:46.427592 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:46.427747  1431 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:46.487078 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.507804 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:46.575482 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:46.585496 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.601719 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:46.620019 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.628784 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.643885 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.681160 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.698767 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.725379 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:46.794955 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:46.809265 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.839895 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.845723 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.856290 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:46.976606 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:46.981513 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:46:46.991722  1741 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:46:46.991894  1741 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:46.992589  1741 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:46.992995 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:46.993773 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:46:46.993953 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:46.994359 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:46.994453 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:46:46.994689  1741 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:47.041738 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:47.081990 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:47.134444 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:47.134747 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:47.211944 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:47.265537 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:47.317126 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:47.352118 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:47.409895 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:47.425858 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:47.451314 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:47.478587 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:47.524483 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:47.527804 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:47.554872 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:47.589540 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:47.625290 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:47.642298 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:47.656098 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:47.669212 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:47.733762 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:47.774770 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:47.785534 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:47.791697 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:47.902122 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240502 20:46:47.903034 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:47.959399 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:47.966931 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:48.044288 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.086256 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:46:48.101044 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:46:48.135421 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:48.199838 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:46:48.203200 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:48.214176 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 141 similar messages]
I20240502 20:46:48.267506  2327 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:48.267684  2327 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:48.268271  2327 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:48.268829 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:48.269145 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:46:48.269630 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:48.269894 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:48.269963 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:48.270153  2327 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:48.328724 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.369257 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.369421 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:48.374241 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:48.401357 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.453687 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:48.538776 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.575181 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.612987 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.628907 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:48.686002  2327 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:48.686184  2327 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:48.686818  2327 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:48.687191 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:48.687407 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:46:48.688413 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:48.688540 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:48.688592 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:48.688843  2327 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:48.708581 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.747537 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:48.750694 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.773236 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.774313 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.785697 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:48.787272 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.824381 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.882632 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.885818 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.893198 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:48.904609 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.934105 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:48.939545 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:48.943701 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.078176 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.098001 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.105427 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.127216 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.172017 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.186904 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.222472 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:49.233811 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:49.271332 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.324746 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.386812 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.391546 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:49.436161 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:49.465727 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.468065 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:49.648495 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.656100 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.669220 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:49.681124 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:49.760624 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:46:49.782933  2480 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:49.783157  2480 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:49.783984  2480 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:49.785192 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:49.785591 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:46:49.786118 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:49.786747 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:49.786814 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:49.787168  2480 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:46:49.798707  2480 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:46:49.798868  2480 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:46:49.799338  2480 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:46:49.800804 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:49.800911 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:49.801386 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:46:49.801486 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:46:49.801705  2480 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:49.829617 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:49.836817 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:49.836838 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:49.867579 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:49.933629 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:49.968139 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:50.021468 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:50.034139 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:50.059747 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:50.064394 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:50.068898 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:50.071101 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:50.102085 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:50.216780 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:50.236407 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:50.249059 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:50.301898 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:50.363991 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:50.438696 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:46:50.442145  2664 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:50.442478  2664 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:50.443240  2664 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:50.443804 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:50.443994 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:46:50.444638 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:50.444801 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:50.444859 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:50.445021  2664 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:50.527586 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:50.533648 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:50.580132 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:50.721930 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:50.750659 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:50.782678 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:46:50.792546 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:50.833807 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:50.882447 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:50.904322 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:50.947644 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:50.998744 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:51.011826 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:51.018157 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:46:51.046598  2571 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:46:51.046829  2571 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:51.047801  2571 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:51.048295 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:46:51.048872 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:46:51.049703 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:51.049842 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:46:51.049978 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:51.050084  2571 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:51.061347 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.194523 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.206311 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.214387 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.246790 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.294811 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.304080 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.306798 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:51.324960 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:51.334581 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.349504 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.358656 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:51.360781 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.389318 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:51.414780 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.424551 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.436043 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:51.493309 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.564678 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.567574 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.626437 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.687038 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.733229 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.769309 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.781293 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:51.797643 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:51.822297 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:51.841691 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:51.861732 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:51.925428 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:51.978899 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:52.001116 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.007870 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.031761 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.247773 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:52.263442 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.267329 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:52.280654 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.288638 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:52.380012 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:52.402858 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:52.440752 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.444267 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.485544 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:52.506351 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:52.531898 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:52.561758 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.563772 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.564352 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.577296 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:52.617429 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.621753 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:52.697423 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:52.713922 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:52.734831 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:52.813266 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:52.935428 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:46:52.943822 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:52.946437 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:52.984542 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.060745 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:46:53.062492 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:53.126201 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:46:53.128265 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:46:53.170078  1576 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:53.170230  1576 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:53.170958  1576 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:53.171334 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:53.171532 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:46:53.172477 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:53.172637 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:46:53.172703 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:53.172907  1576 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:53.217756 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:46:53.236588 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 20:46:53.281880 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.299233 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:53.358901 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.368496 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.429360 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:53.440340 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:53.462779 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:53.490653 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.535005 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.564095 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:53.657011 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.675397 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.679140 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:53.690232 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.693545 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.750718 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.757951 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.761273 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.761289 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.797830 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.798673 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.804340 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.845721 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:53.881942 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:53.903954 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:53.989190 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.055060 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:54.059306 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.085565 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.095717 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:54.124027 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:54.196038 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.255636 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.258298 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.338228 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.373270 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:54.400162 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.408918 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.480815 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:54.495888 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.503983 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:54.532377 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.549160 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:54.555416 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 20:46:54.617656  3028 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:46:54.617902  3028 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:46:54.618966  3028 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:46:54.619920 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:46:54.620441 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:46:54.620532 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:54.621901 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:54.622558 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:54.622656 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:46:54.622870  3028 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:46:54.624773 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:54.754406 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.867709 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.905036 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.918258 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:54.928864 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:54.943871 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:54.967057 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:55.010520 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:55.021570 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:55.054505 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:55.096462 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:55.096489 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:55.165174 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:55.167241 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:55.177207 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:55.192034 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:55.193409 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:55.236749 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:46:55.360666  1576 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:55.360846  1576 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:46:55.361783  1576 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:46:55.362218 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:55.362581 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:46:55.363431 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:55.363556 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:55.363648 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:46:55.363837  1576 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:55.371855 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:55.376121 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:55.417129 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:55.437934 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:55.485020 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:46:55.500509 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:55.518880 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:55.549716 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:55.583549 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:55.720166 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:55.728157 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:55.848659 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:55.903824 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.033164 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:56.054605 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:56.054601 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.068584 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.069247 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.084578 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.084578 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.108779 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:56.123071 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:56.145582 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.160926 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.261165 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.282244 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:56.307494 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.345513 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.349669 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.360894 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.368703 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.489622 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.505872 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.519433 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.532809 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:56.575661 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:56.583321 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:56.658843 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.723641 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:56.790262 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:56.822991 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.836052 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.836300 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.855456 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.897825 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.908267 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:56.946659 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:56.969620 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:57.024828 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.140033 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:57.149045 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:57.183920 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:57.206986 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:57.242368 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:57.291131 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:46:57.293274  3598 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:46:57.293447  3598 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:46:57.294034  3598 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:46:57.294517 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:46:57.294697 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:46:57.295857 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:57.296003 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:46:57.296075 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:46:57.296340  3598 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:46:57.333570 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.412848 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.440984 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:57.468772 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.469161 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:57.497211 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.515056 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:57.603170 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:57.618198 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.642930 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:57.678985 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:57.712002 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:57.750952 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.768193 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:57.773008 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.807688 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.841686 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.858479 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:57.882799 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:46:57.934327 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:57.943418 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 184 similar messages]
W20240502 20:46:57.950919 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:57.997201 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:58.070930 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.115893 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:58.216917 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:58.242750 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 20:46:58.247174 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.259001 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:58.316886 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 20:46:58.318400 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 20:46:58.368674 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.368851 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:58.390486 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.418118 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.426566 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.499063 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.518456 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:58.533354 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.551388 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.557149 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:58.639068 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.769591 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:58.794363 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:58.805344 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:58.834695 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.862766 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.886018 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.947819 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:58.973196 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:59.011830 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:59.017958 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.030097 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.052165 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.060843 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.098901 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.103467 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.138999 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.198048 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.237522 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.255600 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.277302 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.296593 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:59.313843 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:59.337586 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.376085 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:59.377804 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.434110 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:59.532951 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:46:59.664022 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.689285 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:46:59.710816 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.749889 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.787951 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:46:59.825794 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.841387 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:59.850044 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:46:59.864759 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:46:59.887822 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:46:59.960234 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:00.061964 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.078807 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.099428 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:47:00.099727 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.169919 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.232575 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:47:00.301723 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:00.308600  4344 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:00.308771  4344 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:47:00.309353  4344 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:47:00.309752 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:47:00.309949 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:47:00.310931 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:00.311074 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:00.311128 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:47:00.311291  4344 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:00.331059 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:47:00.351879 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.360981 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:47:00.362010 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:00.387563 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:00.403240 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:00.460184 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:47:00.470999 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.492611 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:47:00.559470 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:00.662056 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.701820 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.715579 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.818871 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:00.824717 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:00.846623 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.870816 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:00.895685 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.913369 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:00.925976 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:00.949623 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:00.996328 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.027324 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.038168 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.073566 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:47:01.169653  1576 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:01.169819  1576 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:47:01.170394  1576 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:47:01.171665 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
W20240502 20:47:01.171785 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:01.172034 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:01.171993 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
I20240502 20:47:01.172132 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:01.172339  1576 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:01.172422 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.276304 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.289536 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:01.302858 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:01.336133 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.354017 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.354730 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:01.400859 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.425808 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.441164 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.449751 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:01.462854 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:01.464812 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.519368 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.530927 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.575124 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.599047 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.622603 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.689671 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.696451 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.729454 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:01.739521 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.741850 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:01.742990 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:01.872758 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.887542 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:01.937440 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:01.975338 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:02.109778 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:02.121029 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:02.124294 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:02.162966  4343 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:02.163148  4343 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:47:02.163712  4343 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:47:02.164276 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:47:02.164453 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:47:02.164986 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:02.165113 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:02.165156 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:47:02.165277  4343 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:02.186317 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:02.352339 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:02.448586 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:02.451511 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:02.459550 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:02.566908 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:02.576141 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:02.580960 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:02.591765 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:02.605197 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:02.613148 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:02.614236 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:02.626060 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:02.657853 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:02.678452 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:02.731799 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:02.743602 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:02.846504 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:02.867743 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:02.901806 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:02.902945 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:02.909427 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:02.943733 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:02.943868 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:47:02.944473 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:02.945983 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:47:03.019049 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:03.068243 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:03.138254 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.146798 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.161024 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:03.244824 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 20:47:03.259585 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.279511 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.320786 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 20:47:03.329485 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:03.348306 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:47:03.353331 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:03.356837 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.360363 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.408464 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.466055 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:03.516392 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.564929 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:03.573190  4343 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:03.573415  4343 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:47:03.574621  4343 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:47:03.574844 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:47:03.575033 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:47:03.575544 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:03.575660 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:03.575907 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:03.576231  4343 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:03.610985 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.634101 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.683563 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.741963 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:03.763446 26797 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: ChangeConfig RPC to 127.19.66.130:43307 timed out after 30.000s (SENT)
W20240502 20:47:03.763769 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:03.764218 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.765192 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:47:03.767053 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.822898 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.864126 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.865859 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.898295 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.912977 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:03.912997 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:03.991039 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.017180 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:04.036129 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.039986 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:04.057961 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.078744 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.089651 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.141676 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.196297 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:04.211747 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.278995 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.293375 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:04.306588 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.325320 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.329456 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:04.343192 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.515297 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:04.627789 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:04.737447 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:04.788540 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:04.809494 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:04.831305 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.907524 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:04.914820 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:04.984282 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:47:04.993754 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:05.004169 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:05.005913 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:05.104774 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.138919 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:05.144546 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:05.196907 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:05.214744 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:05.241484 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:05.367617 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.371140 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.377424 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.413455 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:05.420114 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:05.434255 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:05.480217 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:05.489005 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:05.523579 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.529575 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.543437 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:47:05.582118  5067 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:47:05.582327  5067 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:47:05.582919  5067 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:47:05.585264 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:05.585530 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:05.585608 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:47:05.585829  5067 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:05.623064 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:47:05.716678 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.764904 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:05.765697 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.798343 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.821748 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.826447 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:05.872581 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:05.911948 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.971058 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:05.982758 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:06.000983 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.005666 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:06.017031 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.059195 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:47:06.075078 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.094352 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.156055 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.215092 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:06.223241 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.320973 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.336243 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.375142 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:06.387288 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.392212 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.417953 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.475731 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:06.493702 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:06.516417 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.535048 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.559149 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.567562 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.599308 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:06.600154 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.602167 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.640142 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.664433 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.671121 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:06.709434 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:06.858691 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:06.974007 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:07.019868 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.048914 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.065737 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:07.110169 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:07.135805 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:07.299271 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:07.350823 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.460227 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.482267 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.523288 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:07.554719 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:07.569825 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:07.594274 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:07.611771 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:07.638049 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:07.652422 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.674356 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:07.729588 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.746193 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.790134 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:07.863435 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.884316 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.952759 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:47:07.953547 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:07.955641 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:08.001380 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.025827 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.049438 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.052666 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.078184 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:08.078549 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:08.094938 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:47:08.125604 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.126209 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.130110 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:47:08.131314 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:08.228505 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.238211 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.241137 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.262625 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.301569 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.308046 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.324029 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:47:08.352715 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:08.383066 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.405161 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:47:08.426708 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 20:47:08.451403 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.481505 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:08.521430 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.534917 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:08.542464 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.585166 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:08.622489 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.654001 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:08.679489 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.821049 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:08.878332 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.900655 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.957368 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.982800 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:08.999009 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.028132 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.049266 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:09.074138 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.075377 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.083276 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:09.124203 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:09.142750 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:09.209371 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.270584 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.309284 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.401551 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:09.402565  5646 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:47:09.402760  5646 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:47:09.403980  5646 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
W20240502 20:47:09.404256 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 20:47:09.404247 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:47:09.405175 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:09.405470 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:47:09.406014 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:47:09.406282 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:09.406303  5646 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:47:09.413233 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:09.428948 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:09.507572 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:09.540565 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:09.543768 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.687548 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.704715 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:09.739779 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.759954 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:09.829731 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:09.921561 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:09.963527 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:09.963673 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:10.034719 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:10.105386 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.118367 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:10.121827 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:10.128407 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:10.135591 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.179358 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.191145 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:10.220777 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.227771 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:10.246255 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:10.303956 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.319038 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:10.415174 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:10.421048 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:10.510823 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:10.520412 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:47:10.536062 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:10.575471 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.576397 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.583411 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:10.619866 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.658651 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.678303 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.744968 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.759673 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.764752 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.788192 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.838316 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.859549 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:10.896203 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.909188 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:10.913233 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:10.922219 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:10.987594 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:11.130383 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:11.153862 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.154062 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.289873 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.294317 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:11.341401 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:11.346025 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.350360 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.470468 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.478303 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.479851 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:11.487733 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:11.491073 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.550606 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.588641 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.602619 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:11.607235 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.680889 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.700143 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.805670 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.843418 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.859984 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.878399 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:11.898276 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:11.919514 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:11.948575 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:12.089721 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:12.093387 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:12.230201 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.240614 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:47:12.324712 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.354519 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:12.357151 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.392364 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.434734 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:12.499215 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:12.533442 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.563745 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:12.599474 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:12.622848 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.636538 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:12.642136 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.651654 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:12.700259 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.738139 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:12.752957 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:12.766754 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.779493 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:12.796741 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:12.797713 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:12.847986 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.885485 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:12.952734 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:12.978792 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240502 20:47:12.986976 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:13.013046 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:13.109930 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:47:13.113626 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:47:13.133342  6015 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:13.133535  6015 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:47:13.134436  6015 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:47:13.134683 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:47:13.134868 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:47:13.137229 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:13.137400 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:13.137446 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:47:13.137646  6015 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:13.143954 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.153947 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:13.221611 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.265278 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.286824 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.299060 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:13.311777 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.316005 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.326892 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 20:47:13.407150 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:47:13.443665 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.444239 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 20:47:13.459394 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.473840 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.473807 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.492301 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:13.510254 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:13.528131 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:13.595705 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.687427 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:13.690078 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.723202 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.729410 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:13.751786 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.798585 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.800585 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.808861 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:47:13.834440  6015 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:13.834656  6015 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:47:13.835481  6015 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:47:13.836210 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:47:13.836486 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:47:13.837412 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:13.837580 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:13.838055 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:13.838122 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:13.838670  6015 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:13.878736 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:14.020943 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:14.033671 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:14.038053 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:14.067943 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:14.078055 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:14.099987 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:14.162344 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:14.172938 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:14.193259 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:14.355170 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:14.375528 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:14.393745 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:14.517688 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:14.550272 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:14.567330 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:14.605657 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:14.618244 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:14.677722 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:47:14.733565  6015 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:47:14.733757  6015 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:47:14.734450  6015 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:47:14.736043 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:14.736258 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:14.736387 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:47:14.736909  6015 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:14.804154 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:14.913945 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.022533 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.022578 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.029515 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.037251 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:15.043509 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.055189 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:15.103011 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.108698 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:15.141706 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.184046 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.200029 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.236135 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:15.244380 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.261646 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.302323 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.305732 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.323371 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.352450 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:15.454739 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:15.466017 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.475075 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.480784 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.500813 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.520377 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:47:15.564208 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.567905 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:15.600574 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.610110 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:47:15.629920  6015 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:47:15.630090  6015 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:47:15.630736  6015 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:47:15.631103 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:47:15.631693 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:47:15.632639 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:15.633996 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:15.634073 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:15.634302  6015 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:47:15.647392 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.668339 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:15.750417 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.850312 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.881764 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:15.903357 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.920250 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:15.932888 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:16.014911 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.040385 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.057233 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:47:16.080260  6911 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:16.080500  6911 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:47:16.081255  6911 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:47:16.081920 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:47:16.082118 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:47:16.083127 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:16.083216 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:16.083314 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:16.083498  6911 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:16.099192 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:16.114533 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:16.129320 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.184504 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:16.264487 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.272841 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.288182 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.326714 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.341691 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:16.378134 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.386016 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.422343 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:16.458318 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.469272 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.519634 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.580502 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:16.585152 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:16.670899 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.677155 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.714799 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:16.776011 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.856323 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:16.902019 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.908833 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:16.911260 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:16.975450 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:17.088686 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.168498 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:17.181197 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.196409 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:17.309600 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.351253 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.406986 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:17.425333 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:17.426224 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.449417 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.457449 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:17.463398 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.482168 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:17.551476 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:17.569196 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.578116 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.583642 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:17.629801 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.654799 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:17.716028 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.736169 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.763659 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:17.789541 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:17.838809 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:17.918814 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:47:17.936784 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:17.948751 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:17.958585 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:17.974239 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:17.978268 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:17.979130 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:47:17.979992 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:17.980332 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:18.069597 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.090734 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.181686 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.182418 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.209285 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.229018 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.257040 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.327615 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.334185 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 20:47:18.335355 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:18.349277 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:18.388414 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:18.446636 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:47:18.449440 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.470904 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:18.490456 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 20:47:18.498451 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:18.504143 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.514755 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.574492 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.781519 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:18.784369 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.810916 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.821825 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:18.825516 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.829970 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:18.858893 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 20:47:18.893235  6911 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:18.893503  6911 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:47:18.894253  6911 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:47:18.894675 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:47:18.894929 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:47:18.895676 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:18.895926 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:18.896056 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:18.896270  6911 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:18.955355 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:18.970552 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:18.989508 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.000666 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.032078 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.039659 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.041383 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:19.102705 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.152025 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.180709 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.248713 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.350600 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:19.410666 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:19.449105 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:19.463373 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:19.466051 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:19.567314 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.653954 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.664319 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.672694 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.728391 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.845387 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.891817 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:19.923763 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:19.924826 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:19.931486 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:19.960860 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:19.974900 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:20.001154 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:20.026988 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:20.030619 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:20.049052 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:20.051039 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.128602 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:20.206072 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:20.237934 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:20.238567 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:47:20.327324 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:20.352687 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:20.446267 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.462493 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:47:20.507879 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:20.520045 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.542439 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:20.556851 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:20.573666 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.588433 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.629446 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.632525 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.665102 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.729346 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.738996 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.744822 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.754302 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.841332 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:20.859618 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.875911 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.887370 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:20.935374 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:20.965960 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:20.969974 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.034929 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:21.071262 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:21.077437 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.079007 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.203317 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.220651 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.364522 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.372340 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.374920 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:21.390594 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.393831 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:21.407323 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.423954 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.430030 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:21.512426 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:21.544623  7092 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:21.544826  7092 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:47:21.545449  7092 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:47:21.546046 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:47:21.546327 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:47:21.547171 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:21.547314 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:21.547405 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:21.547607  7092 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:21.562824 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:21.604380 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.617599 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:21.629626 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.666458 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.691568 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.714941 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:21.918674 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:21.968542 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.010509 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:22.039671 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:22.044780 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:22.052525 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.094692 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.200078 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:22.296723 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:22.305532 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.347507 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:22.368795 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.397224 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.435889 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.441375 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:47:22.473779  8034 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:22.473959  8034 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:47:22.474514  8034 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:47:22.475064 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:47:22.475267 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:47:22.476115 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:22.476418 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:22.476552 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:47:22.476794  8034 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:22.482983 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.554442 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:22.596980 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:22.623903 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.624225 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:22.630743 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.687712 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.703837 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:22.777936 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.794423 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:22.813501 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:47:22.839244 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:22.850333 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:22.891700 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:22.981882 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:23.050994 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:23.077580 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.081049 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.081321 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:47:23.128119 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.145488 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.150048 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:23.162103 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.169776 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.191812 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.195261 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:23.254760 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.350948 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:23.356979 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.366161 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 20:47:23.392129 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.407567 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:23.452585 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.462467 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:47:23.484093 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.518344 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 20:47:23.519027 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:23.532882 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.538225 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.566257 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.570758 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.573164 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.660707 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.747998 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.753106 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:47:23.773286  8315 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:47:23.773553  8315 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:47:23.774359  8315 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:47:23.774982 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:47:23.775377 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:47:23.776222 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:23.776325 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:47:23.776436 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:23.776510  8315 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:47:23.807855 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:23.865285 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.866235 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:23.909539 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:23.945748 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.023972 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:24.034827 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:24.055824 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.059965 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.129086 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.175269 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.196590 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.272461 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.317652 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.328642 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:24.344149 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:24.354408 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:24.371577 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.460378 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.486397 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:24.524473 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:24.749862 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:24.781221 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:24.832110 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.842599 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:24.896564 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.899081 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.907346 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.932703 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.950029 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:24.950378 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.043885 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:25.049424 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:25.051326 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:25.075202 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:25.091480 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.104522 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:25.146015 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.185488 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:25.237390 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:25.257745  8314 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:25.257982  8314 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:47:25.258881  8314 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:47:25.259403 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:47:25.259618 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:47:25.260627 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:25.260902 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:25.261004 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:25.261222  8314 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:25.320427 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.330454 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:25.333428 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:25.444432 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:47:25.463156 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.532786 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.550338 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.590997 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:25.633535 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:25.670923 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.714598 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:25.726775 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.750882 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.761341 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:25.774780 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.804924 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:25.886571 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.943375 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.951030 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:25.988360 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:26.000281 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.058909 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.070993 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.117399 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.154431 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.155944 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.183358 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:26.203202 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:26.225972 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.259819 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:26.261001 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.330652 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:26.366247 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:26.398800 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.497691 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.509166 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.536696 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.548521 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.557626 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.718660 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:26.756223 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.766480 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:26.782953 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:26.824934 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.843431 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:26.884428 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.885973 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.911489 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:26.935770 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:26.961853 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.136492 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.175597 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:27.175801 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:27.182543 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:27.203720 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.217255 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:27.456115 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:27.463675 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:27.482653 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.497944 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:47:27.553478  8976 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:27.553792  8976 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:47:27.554473  8976 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:47:27.554834 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:47:27.555003 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:47:27.556845 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:27.557253 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:47:27.558043 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:47:27.559088  8976 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:27.601846 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.602708 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.609611 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.619544 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:27.640635 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:27.646133 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.667234 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:27.704457 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:27.721201 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.775864 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.800048 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.861052 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:27.913048 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:27.928606 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:27.951272 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:27.954531 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:28.013218 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:28.041287 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:47:28.068572 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.105316 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:28.159583 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:28.165270 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 184 similar messages]
W20240502 20:47:28.170370 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.179515 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:28.186841 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.213054 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.239084 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.302137 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.318970 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.381464 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.466113 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:47:28.466995 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.501906 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:28.525303 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 20:47:28.575237 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.591699 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.603811 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:28.606125 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:28.615392 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.677556 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.703758 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:28.715342 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.739739 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.764310 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:28.776515 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:47:28.777232 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.811183 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.947841 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:28.966195 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.977248 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:28.985443 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:29.041160 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.118678 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:29.148394 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.250578 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.251693 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:29.256196 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:29.262661 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.324245 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.380642 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:29.408787 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.469964 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:29.531239 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.600081 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:29.610661 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.627218 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.654475 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.714186 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:29.718578 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:29.838197 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:29.839143 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:29.874243 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:47:29.907232  9208 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:47:29.907438  9208 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:47:29.908015  9208 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:47:29.909336 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:29.909440 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:47:29.909529 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:47:29.909749  9208 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:29.961980 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:29.969229 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:29.995024 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:30.048983 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.064101 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:30.078015 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:30.096587 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.106030 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:30.144621 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:30.177367 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:30.186483 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:30.198364 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:30.364784 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:30.371740 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:30.415465 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:30.417444 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:30.450866 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:30.460381 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.462466 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:30.489732 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:30.501267 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:47:30.546082 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.595821 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.623410 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.713994 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.733920 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.738750 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.771001 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.824470 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.834545 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.881320 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:30.890967 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:30.911142 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:30.926843 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:30.962353 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:30.990356 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.007025 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.020853 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:31.045943 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.049490 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.172684 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.199494 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.328924 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.337740 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.353057 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:31.423928 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:31.445806 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.457026 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:31.468976 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:31.489683 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.495925 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.579063 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:31.713621 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.715303 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.824723 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:31.949342 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:31.954408 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:32.000903 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.023581 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:32.029253 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.060215 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:32.074273 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.076440 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:32.098991 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.196827 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.201882 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.219517 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.261232 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:32.273777 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.279340 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.368151 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:32.377462 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.397032 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:32.431293 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:32.488654 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:32.489964 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:32.496582 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:32.515856 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:32.607410 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:32.697837 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.712859 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.752149 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.786623 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.808903 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.849512 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:32.915273 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:32.926494 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:32.972082 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:33.070569 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:33.087819 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:33.109330 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:47:33.127607 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.128041 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.182577 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.229372 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.266429 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:47:33.305099 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.321673 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.331712 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:33.346599 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:33.363715 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.389233 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.413614 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.429733 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:33.438771 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.476413 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 20:47:33.490268 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.505863 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.542979 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 20:47:33.543889 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:33.554657 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.643599 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.644486 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:33.668298 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.692730 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.740154 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.805138 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:33.820277 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:47:33.821398 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.839615 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:33.859797 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:33.973192 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:34.012112 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.051427 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.073516 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:34.086796 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.197191 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.218328 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.239701 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:47:34.254194 10050 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:34.254357 10050 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:47:34.254984 10050 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:47:34.255757 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:47:34.255944 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:47:34.256280 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:34.256565 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:34.256686 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:34.256870 10050 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:34.298942 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:34.358217 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.368230 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:34.405006 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:47:34.409555  9771 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:34.409771  9771 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:47:34.410389  9771 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:47:34.410920 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:47:34.411087 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:47:34.412006 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:34.412127 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:47:34.412182 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:47:34.412360  9771 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:34.454381 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:34.456763 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.483603 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.497011 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.547504 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.552866 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.584869 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:34.661360 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:34.711997 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.771075 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:34.816403 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.857471 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:34.861620 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.871600 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:34.891572 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:34.919250 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:34.950002 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:35.069198 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.130673 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:35.137215 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:35.148566 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:35.175829 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:35.245002 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:35.267875 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.328429 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:35.334187 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:35.371134 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:35.421392 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:35.482709 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.484345 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:35.531540 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:47:35.532157 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.536247 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:35.571116 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:35.630373 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.675881 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.675997 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.712409 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:35.743345 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.746762 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:35.772854 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:35.841627 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.868486 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.939795 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:35.978156 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.025604 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.051431 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.073206 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:36.139969 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:36.143278 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:36.160897 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:36.187793 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.206097 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.209910 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:36.219151 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.235495 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.280885 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.296551 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.307094 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.329759 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.419590 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.499507 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:36.521970 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.583403 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:36.589411 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.606559 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.697490 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:36.727216 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:36.748193 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:36.763444 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.889135 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:47:36.890080 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.932722 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:36.983191 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:37.034020 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.043743 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.085034 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:37.107445 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.170594 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:37.209148 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:37.233897 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:37.253482 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.263175 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.264637 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:37.273691 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.311300 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.326233 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.368110 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:37.510986 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:37.511065 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.588284 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:37.608263 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:37.608331 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:37.621325 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:37.665766 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.666473 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:37.685470 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:37.744578 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.800001 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:37.818370 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:37.839694 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.875139 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.924299 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:47:37.926786 10747 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:37.927193 10747 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:47:37.927940 10747 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:47:37.928762 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:47:37.929006 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:47:37.930039 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:37.930186 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:37.930259 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:47:37.930438 10747 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:37.962621 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:37.999241 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.086639 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:38.123104 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:47:38.176690 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.208535 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:47:38.228224 10519 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:47:38.228431 10519 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:47:38.229190 10519 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:47:38.229571 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:47:38.229909 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:38.230387 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:38.230461 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:47:38.230865 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:47:38.231021 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:38.231019 10519 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:47:38.266376 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:38.282976 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 184 similar messages]
W20240502 20:47:38.342206 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.351534 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.410777 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:38.431113 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.451205 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.483397 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 20:47:38.484222 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.486778 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.486778 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.528198 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:47:38.559509 10846 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:47:38.559695 10846 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:47:38.560175 10846 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:47:38.560726 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 20:47:38.561457 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:38.561568 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:47:38.561655 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:47:38.561817 10846 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:38.579645 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.605211 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.630544 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.633009 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:38.727027 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:38.763972 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.780920 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.822279 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:47:38.823318 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.832569 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:38.833693 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.872766 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:38.881875 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:38.899752 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:38.916970 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.018255 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.046021 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.084904 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:39.125831 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.221951 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:39.272267 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.284531 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:39.327649 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.350899 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:39.353639 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:39.368279 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.482741 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.562589 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:39.667189 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:39.670320 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.679033 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.759775 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:39.762909 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:39.763919 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.798174 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.806192 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:39.830962 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.908823 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.945912 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:39.947341 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:39.947701 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:39.981182 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.097594 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.112300 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:40.143635 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.197288 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.213589 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:40.282358 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:40.284534 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:40.345280 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:40.386075 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:40.436693 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:40.449846 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:40.495357 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.511065 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:40.525200 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:40.580709 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.650395 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.650635 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:40.661573 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:40.680049 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:47:40.709666 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.743155 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.763259 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:40.931409 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:40.933616 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.938457 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.946291 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:40.982335 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.982460 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.988348 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:40.991628 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:41.020753 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.141709 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.176530 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.178118 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.189818 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:41.202814 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.242065 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:41.298300 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.364275 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:41.385501 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.387252 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.397598 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:41.432173 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.499711 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.515812 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.567736 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:41.599933 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.658066 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:41.685796 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.743747 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:41.773705 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.791754 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:41.888644 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:41.931950 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:42.040694 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.091473 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.104419 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.124593 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.168784 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:42.174127 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.181349 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:42.227288 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:42.280463 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:42.283041 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.291469 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.321245 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:42.327286 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:42.333276 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.351828 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.434370 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:42.473260 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.608877 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:42.626587 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:42.638676 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:42.647953 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:42.652976 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:42.793574 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.852499 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:42.866264 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:42.892508 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:42.909894 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:43.028636 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.061935 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:43.074298 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.082486 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:43.106177 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:43.112782 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:43.136611 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.142359 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:43.217350 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.232383 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.270613 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:43.303546 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:43.341801 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 178 similar messages]
W20240502 20:47:43.342730 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.343976 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.396834 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:43.403061 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:47:43.539886 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.541458 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.556574 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 20:47:43.557303 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:43.563006 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 20:47:43.576573 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:43.689764 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.703598 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:43.722316 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.739003 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.751865 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.762801 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.802207 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:43.807724 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.808084 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.808259 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.862371 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:47:43.906468 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:43.956975 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:43.965344 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.005497 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.026576 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.037482 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:44.106923 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.131742 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:44.151098 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.263689 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.310912 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.329203 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:44.358796 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.371156 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:44.471585 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.519274 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:44.545428 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:44.598740 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:47:44.609436 11637 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:47:44.609673 11637 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:47:44.610531 11637 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:47:44.611027 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:47:44.611280 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:47:44.611914 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:44.612003 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:47:44.612183 11637 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:47:44.612249 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:44.683724 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.713160 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:44.716400 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.737176 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:44.755385 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.836861 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.856365 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.874958 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:44.912432 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:44.915575 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:44.943598 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:44.961264 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:44.969291 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:45.060448 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.075085 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:45.180285 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:45.192869 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.290271 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:45.302234 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:45.360819 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:45.362586 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.418715 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:45.448571 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:45.457350 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:45.457640 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:45.458621 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.463308 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:45.579669 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.590932 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.601162 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:45.644573 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.648563 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.732808 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:45.740087 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.786761 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.805347 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:45.815405 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:45.929649 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:45.935446 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:47:45.983232 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:46.030124 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.055150 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:46.108798 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.180685 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.183102 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:46.207182 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:46.226662 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.339105 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.341012 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.359005 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.402101 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.403016 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:46.427142 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:46.430197 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.458438 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:46.467520 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.474020 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.567289 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.567652 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:46.579484 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.650871 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.662282 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.682600 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:46.724749 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.767371 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.780153 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.836053 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:46.905689 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:46.933461 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:46.978725 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:47.034519 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:47.079852 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.084549 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.155120 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.251705 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:47.253501 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.278831 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:47.372067 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:47.392027 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:47.514125 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:47.517884 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:47.564486 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.620781 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:47.621619 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.624621 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:47.653134 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.684185 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:47.782187 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:47.794415 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.809700 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.815959 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:47.834111 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:47.834798 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.876554 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:47.900205 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:47.918191 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:47.923369 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:47.983668 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.044140 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:48.045938 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:48.129377 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:48.131147 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.212373 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.212571 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:47:48.252264 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.299131 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.317510 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.335170 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.375434 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:47:48.400317 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:48.407222 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:48.429597 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.439056 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:48.483577 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:48.486333 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.571252 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 20:47:48.572052 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:48.590597 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240502 20:47:48.600028 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.650830 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:47:48.758224 12520 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:48.758425 12520 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:47:48.759056 12520 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:47:48.760046 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:47:48.760242 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:47:48.760859 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:48.760950 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:48.760994 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:48.761184 12520 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:48.772159 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.820413 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.824208 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:48.827790 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.864288 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:47:48.865410 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.870163 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:48.879130 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:48.894958 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:48.944960 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:48.949915 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.025655 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.074507 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.083312 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.092381 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:49.112563 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.143410 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.169507 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.204815 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.298125 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:49.319010 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:49.345047 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:49.442528 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.459653 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.482728 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.493913 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:49.521009 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.543385 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.622002 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:49.639024 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:49.674475 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:49.816783 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:49.828817 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:49.873729 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:49.900950 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:50.012789 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:50.018922 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.050169 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.112915 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:50.123775 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:50.132355 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:50.268402 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:50.311393 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:50.356357 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.372598 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:50.385152 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:50.394371 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:50.428202 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:50.433315 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:50.445040 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:50.480743 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:50.514438 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.532675 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.562837 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:50.580994 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:50.587484 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.677615 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.731240 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.736307 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:47:50.738970 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:47:50.819660 12894 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:50.819871 12894 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:47:50.820475 12894 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
W20240502 20:47:50.820593 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:47:50.821321 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:47:50.821580 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:47:50.822145 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:50.822288 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:47:50.822342 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:47:50.822504 12894 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:50.830078 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:50.842579 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.860384 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:50.885651 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.899565 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.932152 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:50.967554 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:50.990319 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.022300 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:51.028709 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:51.143002 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.195215 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.227056 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:51.289433 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:51.327754 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.441866 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.447366 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.467096 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:51.479842 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.490298 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.497964 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.499122 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.517877 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.548380 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:51.590216 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.622555 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.628566 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:51.638499 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.688446 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:51.701371 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:51.760368 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:47:51.811146 12102 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:47:51.811367 12102 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:47:51.812193 12102 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:47:51.813266 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:47:51.813580 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:47:51.814049 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:51.814206 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:47:51.814321 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:47:51.814494 12102 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:51.890204 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:51.904937 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.004405 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.005419 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.031045 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.032135 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.091737 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:52.098668 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.130672 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:52.185916 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:52.209022 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:52.272774 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:52.292435 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:52.332115 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:52.492337 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.539904 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:52.569475 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:52.575490 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:52.601833 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.606498 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.658610 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:52.679275 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:52.777284 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:52.778390 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:52.808713 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.885277 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.909142 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.934517 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:52.967854 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:52.970186 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:53.040453 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:53.044229 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:53.055459 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:53.084405 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.091627 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:53.097259 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.131366 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.139537 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.174038 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.185988 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:53.205489 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.212435 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:53.213316 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.238500 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:47:53.271004 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.336138 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.345291 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.376238 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240502 20:47:53.377467 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.441255 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:53.470233 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:53.558463 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.561651 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:53.591991 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:47:53.615723 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240502 20:47:53.619477 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.687427 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.758934 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:53.773120 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:53.803653 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.815380 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.831593 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.910686 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:47:53.911619 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:53.982801 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.017017 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:54.044931 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.050500 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:54.063266 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.090185 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:54.093295 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.223421 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.223523 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.247934 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.281414 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:54.349736 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:47:54.356204 12847 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:47:54.356406 12847 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:47:54.357025 12847 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:47:54.358857 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:54.360041 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:47:54.360127 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:47:54.360374 12847 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:47:54.424717 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.503317 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:54.513404 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.529762 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.586678 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:54.648114 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.649283 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.692643 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:54.727885 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:54.731351 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:54.740826 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.784037 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:54.814949 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:54.893950 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:54.945523 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:54.998525 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.113263 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:55.117178 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:55.216219 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:55.216924 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.223353 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:55.254263 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:55.264551 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:55.265034 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:55.320853 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:55.332723 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:55.430917 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:55.474432 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.478652 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.543768 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:55.550107 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:55.603564 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:55.615725 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:55.627010 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.652408 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.660434 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:55.666695 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.698870 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.745321 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:55.748672 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.796722 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.812937 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:47:55.832960 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:55.867710 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:55.946924 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:56.070541 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.070902 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.076380 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:56.095161 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.119187 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.126622 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.157841 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:56.187500 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.213354 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:56.230929 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.263145 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.374011 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:56.430795 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.436439 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:56.508857 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:56.514045 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.554221 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.614078 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.614565 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.621186 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.663146 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.684942 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.696815 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:56.719708 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.779479 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:56.856837 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:56.864567 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:56.940753 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:56.977048 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.068378 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:57.086242 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.259083 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.274945 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:57.285679 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.288269 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:57.290383 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.300177 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:57.336232 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:57.342036 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.369946 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:57.445488 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:57.604413 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:57.617640 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.618255 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:57.750517 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:57.767568 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:57.774289 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.803092 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:57.812211 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.813310 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.838106 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.855232 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:57.884513 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:57.888918 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.922878 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:57.953466 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:57.978519 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:57.989888 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:58.014968 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.144109 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.145972 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.185923 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:58.193048 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:58.254676 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.282272 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.342684 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:47:58.360538 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:58.361884 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:58.373806 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.402873 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:47:58.436184 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:58.458065 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:58.505329 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.572402 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.586270 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.600777 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240502 20:47:58.616039 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240502 20:47:58.616883 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.635946 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.735734 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.747588 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.790701 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:58.843705 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.854422 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:58.857903 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:58.860047 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:58.867019 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.944541 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:58.977684 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:58.983963 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:47:58.994823 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.145498 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.150434 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.235687 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:47:59.257861 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.298377 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.321857 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:59.334990 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:59.401738 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.415385 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:47:59.428447 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:59.488675 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.541877 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.562871 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.685881 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:47:59.698227 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:47:59.738435 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.745316 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:47:59.775905 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.776192 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:47:59.853654 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:00.003331 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.011301 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:00.013756 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:00.085601 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:00.110718 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:00.117748 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:00.155638 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:00.161429 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.201782 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.267128 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.284613 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:00.338644 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:00.354120 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:00.374214 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.386837 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:00.409807 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:00.437438 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:00.460937 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:00.480290 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:00.494336 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.497812 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:00.502705 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.543627 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.654824 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:00.663515 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.711282 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:00.764245 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:00.865867 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:00.875286 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.885522 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.888080 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:00.892326 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.924432 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:00.967526 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:00.979418 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.042054 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.047024 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:01.078953 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.152243 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:48:01.154451 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:01.199570 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.199769 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:48:01.220407 14062 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:48:01.220571 14062 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:48:01.221316 14062 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:48:01.221729 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:48:01.221997 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:48:01.223070 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:01.223155 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:48:01.223349 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:01.223688 14062 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:48:01.287979 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.290338 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.303562 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:01.379912 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.395035 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.446812 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:01.461477 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:01.514394 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.538264 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.555814 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.602854 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.638384 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:01.743584 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:01.852619 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:01.871031 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.880606 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:01.886655 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:02.001688 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.019519 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:02.057663 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.090196 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.145738 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:02.185981 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.194538 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:02.200002 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.229882 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:02.229895 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:48:02.346614 14628 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:02.346746 14628 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:48:02.347218 14628 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:48:02.347913 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:48:02.348105 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:48:02.348620 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:02.348739 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:02.348785 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:48:02.348933 14816 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:02.409464 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:02.418296 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:02.435873 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.525429 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:02.600091 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:02.636948 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:02.687371 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:02.704756 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:02.723793 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:02.742529 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:02.760233 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.783831 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.820652 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:02.871100 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.871640 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:02.888211 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.899914 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:02.915393 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:02.951453 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:02.959252 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.040683 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:03.054250 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:03.062702 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:03.174165 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.185420 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:03.198138 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:03.204999 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.217553 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.243263 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:03.269985 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.273151 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.291342 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:03.324725 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:03.342645 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:03.364321 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.414134 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.431032 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:48:03.443825 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.605284 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:03.630031 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 20:48:03.643070 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 20:48:03.683326 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.713234 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.722007 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:03.766700 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:03.770799 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:03.779961 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.793727 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:48:03.798207 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.857010 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.900475 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:03.936305 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:03.984637 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:48:04.023422 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:04.043715 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.149749 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.161042 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:04.167368 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.173754 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.308714 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.317180 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:04.344595 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:04.371701 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:04.447263 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.451030 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:04.500156 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.590939 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.607302 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:04.610014 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.615375 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.624311 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.735738 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:04.763886 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.777459 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.882067 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:04.898049 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:04.927690 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:04.928292 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:04.983673 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:05.008373 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:05.048453 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.083482 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.192404 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.193876 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:05.260219 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.280982 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.284777 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:05.377369 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:05.492102 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:05.492460 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.498019 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:05.518944 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:05.520380 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:05.528530 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:05.569842 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.583216 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.589450 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:05.624737 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:05.648114 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:05.683099 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.684630 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:05.708336 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.717905 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.770223 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:05.773993 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:05.832782 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:05.918183 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:05.922443 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:05.976922 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.021459 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:48:06.053921 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:06.073235 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:06.110780 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:06.172331 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:06.197171 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.231688 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.242612 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:48:06.280994 15489 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:48:06.281199 15489 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:48:06.281847 15489 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:48:06.282248 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:48:06.282663 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:48:06.283922 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:06.284013 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:48:06.284109 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:06.284193 15489 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:48:06.324892 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.333556 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:06.365007 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.383244 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.391675 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.431937 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.453045 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:06.503540 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.551192 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.623308 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:06.627461 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:06.651968 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:06.748760 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:06.857954 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:06.868242 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:06.905625 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:07.005877 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.052815 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:07.081049 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.090036 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:07.118568 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.146044 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:07.184597 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.255388 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.260879 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.275826 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.367831 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:07.392623 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.394088 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:07.398726 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.399891 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.402683 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:07.442479 15428 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:07.442626 15428 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:48:07.443140 15428 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:48:07.443655 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:48:07.443832 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:48:07.444795 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:07.444916 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:07.445019 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:48:07.445413 15428 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:07.550124 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:07.558157 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:07.576179 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:07.581652 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:07.631096 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.675830 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:07.801676 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.815332 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.820399 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:07.824131 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:07.824934 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:07.830113 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:07.878909 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:07.920495 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:07.961367 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:08.026415 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.049029 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:08.121507 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:08.125684 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.132207 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.141561 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.146147 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.161427 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:08.174929 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:08.322748 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:08.345579 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:08.377187 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.402877 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:08.442636 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:08.446770 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 178 similar messages]
W20240502 20:48:08.462801 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:08.526188 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.527081 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:08.557479 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.561514 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.562346 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:08.592172 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.642031 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.643312 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:48:08.646376 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240502 20:48:08.648689 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240502 20:48:08.649497 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:08.673771 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.684916 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.720431 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:08.788822 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:08.856220 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:08.959934 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:08.992177 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.010800 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.049024 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:09.075289 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:48:09.105011 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.111788 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:09.275646 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.285995 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.294211 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.330415 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:09.339461 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:09.342550 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:09.469036 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.479565 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:09.503858 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:09.565297 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:09.589627 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.637929 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.668769 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:48:09.681375 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.681502 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:09.709420 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.785306 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.793203 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.829581 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:09.852567 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:09.946089 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:09.961477 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:09.966843 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:10.000416 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:10.000468 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:10.011859 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:10.110220 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:10.234083 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:10.253158 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:10.255565 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:10.330312 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:10.338168 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:10.387382 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:10.406317 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:10.407265 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:10.435245 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:10.452059 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:10.452227 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:10.513489 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:10.526782 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:10.544602 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:10.549537 16102 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:48:10.549777 16102 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:48:10.550359 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:48:10.550957 16102 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:48:10.551537 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:48:10.552043 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:48:10.552791 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:10.552968 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:48:10.553349 16102 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:48:10.553166 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:10.563793 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:10.674860 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:10.686923 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:10.694620 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:10.731176 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:10.840418 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:10.855942 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:10.904070 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:10.947088 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:10.950242 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:10.992048 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:11.016054 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.049708 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:11.064384 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.065318 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:11.069661 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.132118 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.187870 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:11.200776 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.209801 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.222193 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.263825 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:48:11.419838 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:11.486055 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:11.514159 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:11.552448 26799 maintenance_manager.cc:419] P e0c668105a124f0eb7227390002cc15e: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033388
W20240502 20:48:11.553534 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:11.569267 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:11.581519 26798 maintenance_manager.cc:643] P e0c668105a124f0eb7227390002cc15e: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.028s	user 0.024s	sys 0.004s Metrics: {"bytes_written":21918,"cfile_init":1,"dirs.queue_time_us":348,"dirs.run_cpu_time_us":300,"dirs.run_wall_time_us":1215,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":908,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":258,"threads_started":2,"wal-append.queue_time_us":223}
I20240502 20:48:11.582757 26799 maintenance_manager.cc:419] P e0c668105a124f0eb7227390002cc15e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10357 bytes on disk
I20240502 20:48:11.583588 26798 maintenance_manager.cc:643] P e0c668105a124f0eb7227390002cc15e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240502 20:48:11.588253 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.591773 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.702958 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.778384 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.845728 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:11.887310 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.901897 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:11.949304 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:11.952315 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:11.968613 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:11.984086 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.005584 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.049626 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.126816 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:12.152586 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:12.155902 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:12.160506 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.244865 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.271253 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.279824 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.302724 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.327267 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:12.367092 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.486901 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:12.490969 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:12.525107 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:12.555948 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.642311 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:12.644798 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:12.650419 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:12.661193 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.691967 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.738416 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:12.848049 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:12.889242 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:12.933785 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.018285 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:13.025516 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:13.029193 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:13.101348 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.101408 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.106469 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.107834 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.118388 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:13.121287 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.123521 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:13.192474 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:13.199306 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.260782 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.265486 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:13.278116 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:48:13.301661 16233 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:48:13.301913 16233 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:48:13.302624 16233 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:48:13.303942 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:13.304049 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:13.304191 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:48:13.304399 16233 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:13.408699 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:13.429944 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.463352 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 178 similar messages]
W20240502 20:48:13.539358 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:13.551206 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.582159 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.596518 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:13.597219 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:13.603714 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.616143 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:13.630033 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:13.642716 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:13.667439 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 20:48:13.668393 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.770365 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240502 20:48:13.785944 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:13.812588 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:48:13.895912 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.919235 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.982795 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:13.991429 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:48:13.991953 16233 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:13.992146 16233 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:48:13.992767 16233 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:48:13.993363 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:48:13.993577 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:48:13.994025 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:13.994568 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:13.994722 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:48:13.994915 16233 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:14.137171 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:14.162024 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:14.184163 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:14.184252 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:48:14.220216 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.261379 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:14.277824 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.308270 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.339262 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:14.348493 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.404253 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.442413 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:14.443729 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.513622 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.561043 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.581049 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:14.615990 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:14.629964 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:14.678332 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:14.720340 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:14.731199 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.781910 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:14.793407 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.824704 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.835883 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.927598 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:14.946687 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:15.015774 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:15.048008 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:15.083243 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:15.091701 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.096566 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:15.172405 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:15.229246 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:15.257189 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:15.261585 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.316017 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.329303 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:15.384542 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.430159 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.431886 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:15.462854 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.478317 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:15.496944 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:15.554713 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:15.587330 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:15.598126 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.690061 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:15.700995 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.710345 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:15.716061 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:15.763943 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.828532 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.846987 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.851071 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:15.903446 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:15.957392 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:15.964144 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:16.029443 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.101867 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:16.126977 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:16.157526 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.171345 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:16.184840 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.205265 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:16.239595 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.240980 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:48:16.283393 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:16.318668 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.385676 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.430166 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:16.511082 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:16.535254 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:16.548285 17033 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:48:16.548586 17033 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:48:16.549589 17033 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:48:16.550356 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:48:16.550518 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:48:16.550822 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:16.551012 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:48:16.551256 17033 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:48:16.551625 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:16.579828 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:16.608765 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.670280 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.740918 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:16.743610 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.769939 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:16.783226 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.839767 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:16.845850 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:16.871371 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:16.872825 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:16.934725 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:17.021317 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:17.022327 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:17.034054 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:17.107708 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:17.115310 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:17.115437 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:17.139788 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(f2dfc0bbbbca4cc28a97480007343424): perf score=0.033347
I20240502 20:48:17.185660 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(f2dfc0bbbbca4cc28a97480007343424) complete. Timing: real 0.045s	user 0.036s	sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":202,"dirs.queue_time_us":380,"dirs.run_cpu_time_us":372,"dirs.run_wall_time_us":1263,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":851,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":396,"threads_started":3,"wal-append.queue_time_us":266}
I20240502 20:48:17.187745 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(f2dfc0bbbbca4cc28a97480007343424): 4925 bytes on disk
I20240502 20:48:17.188699 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(f2dfc0bbbbca4cc28a97480007343424) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240502 20:48:17.228475 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:17.250986 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:17.274544 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:17.307788 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:17.315866 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:48:17.327678 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(d44f271f166b4c288b76689a3e3764d3): perf score=0.033394
I20240502 20:48:17.328130 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(078a1591d9d74e31a0407aeab76911b4): perf score=0.033397
W20240502 20:48:17.345824 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:17.392351 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:17.392985 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(d44f271f166b4c288b76689a3e3764d3) complete. Timing: real 0.065s	user 0.033s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":232,"dirs.queue_time_us":8869,"dirs.run_cpu_time_us":305,"dirs.run_wall_time_us":1266,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":845,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":403,"threads_started":3,"wal-append.queue_time_us":3431}
I20240502 20:48:17.395223 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(d44f271f166b4c288b76689a3e3764d3): 4586 bytes on disk
I20240502 20:48:17.396345 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(d44f271f166b4c288b76689a3e3764d3) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":190,"lbm_reads_lt_1ms":4}
I20240502 20:48:17.414675 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(078a1591d9d74e31a0407aeab76911b4) complete. Timing: real 0.086s	user 0.041s	sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":606,"dirs.queue_time_us":329,"dirs.run_cpu_time_us":362,"dirs.run_wall_time_us":9477,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":821,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":433,"threads_started":3,"wal-append.queue_time_us":433}
I20240502 20:48:17.416947 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(f2dfc0bbbbca4cc28a97480007343424): perf score=0.033420
I20240502 20:48:17.441623 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(f2dfc0bbbbca4cc28a97480007343424) complete. Timing: real 0.024s	user 0.021s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":75,"dirs.run_cpu_time_us":263,"dirs.run_wall_time_us":2831,"drs_written":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4,"lbm_write_time_us":716,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":200,"threads_started":1,"wal-append.queue_time_us":341}
I20240502 20:48:17.442849 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(f2dfc0bbbbca4cc28a97480007343424): 4925 bytes on disk
I20240502 20:48:17.443837 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(f2dfc0bbbbca4cc28a97480007343424) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 20:48:17.444945 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(078a1591d9d74e31a0407aeab76911b4): 4840 bytes on disk
I20240502 20:48:17.445487 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(078a1591d9d74e31a0407aeab76911b4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240502 20:48:17.522192 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:17.528515 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:17.532397 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:17.565598 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:17.650725 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:17.695905 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:17.769446 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:17.791112 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:17.899873 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(d1b9ff3c921d474fa5ad106d1c17ab00): perf score=0.033337
W20240502 20:48:17.909341 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:17.915695 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:17.921820 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(d1b9ff3c921d474fa5ad106d1c17ab00) complete. Timing: real 0.022s	user 0.019s	sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":211,"dirs.run_cpu_time_us":340,"dirs.run_wall_time_us":1829,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":210,"threads_started":2,"wal-append.queue_time_us":284}
I20240502 20:48:17.923158 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(367543df62e048268908eb0d84c14876): perf score=0.033342
W20240502 20:48:17.930740 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:17.935241 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:17.941893 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(367543df62e048268908eb0d84c14876) complete. Timing: real 0.019s	user 0.016s	sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":242,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1840,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":702,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":114,"threads_started":1,"wal-append.queue_time_us":238}
I20240502 20:48:17.942960 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(d1b9ff3c921d474fa5ad106d1c17ab00): 4925 bytes on disk
I20240502 20:48:17.943069 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(b466a750cc6b4382947282d8aa6aeb1f): perf score=0.033349
I20240502 20:48:17.943462 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(d1b9ff3c921d474fa5ad106d1c17ab00) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 20:48:17.944486 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(367543df62e048268908eb0d84c14876): 4586 bytes on disk
I20240502 20:48:17.944927 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(367543df62e048268908eb0d84c14876) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 20:48:17.948083 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(b466a750cc6b4382947282d8aa6aeb1f): perf score=0.033344
W20240502 20:48:17.956697 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:17.963251 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(b466a750cc6b4382947282d8aa6aeb1f) complete. Timing: real 0.020s	user 0.018s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":493,"dirs.run_cpu_time_us":306,"dirs.run_wall_time_us":2157,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":632,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":172,"threads_started":2,"wal-append.queue_time_us":194}
I20240502 20:48:17.964380 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(b466a750cc6b4382947282d8aa6aeb1f): 4840 bytes on disk
I20240502 20:48:17.965023 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(b466a750cc6b4382947282d8aa6aeb1f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 20:48:17.968116 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(b466a750cc6b4382947282d8aa6aeb1f) complete. Timing: real 0.020s	user 0.018s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":321,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1558,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":567,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":200,"threads_started":2,"wal-append.queue_time_us":210}
I20240502 20:48:17.969656 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(b466a750cc6b4382947282d8aa6aeb1f): 4840 bytes on disk
I20240502 20:48:17.970440 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(b466a750cc6b4382947282d8aa6aeb1f) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4}
W20240502 20:48:17.989662 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:18.008221 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.043818 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:18.044775 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.059469 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:18.068011 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:48:18.100638 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(d1b9ff3c921d474fa5ad106d1c17ab00): perf score=0.033390
W20240502 20:48:18.132155 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:48:18.145203 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(d1b9ff3c921d474fa5ad106d1c17ab00) complete. Timing: real 0.044s	user 0.039s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":194,"dirs.queue_time_us":259,"dirs.run_cpu_time_us":267,"dirs.run_wall_time_us":960,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":794,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":297,"threads_started":3,"wal-append.queue_time_us":301}
I20240502 20:48:18.146855 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(d1b9ff3c921d474fa5ad106d1c17ab00): 4925 bytes on disk
I20240502 20:48:18.147850 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(d1b9ff3c921d474fa5ad106d1c17ab00) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240502 20:48:18.149471 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(367543df62e048268908eb0d84c14876): perf score=0.033401
W20240502 20:48:18.162300 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:18.167634 17360 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:18.167667 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(367543df62e048268908eb0d84c14876) complete. Timing: real 0.018s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1532,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":869,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":142,"threads_started":1,"wal-append.queue_time_us":245}
I20240502 20:48:18.167825 17360 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:48:18.168586 17360 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:48:18.169384 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:48:18.169526 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(367543df62e048268908eb0d84c14876): 4586 bytes on disk
I20240502 20:48:18.169536 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
I20240502 20:48:18.170729 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(367543df62e048268908eb0d84c14876) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
W20240502 20:48:18.171347 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:18.171509 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:18.171566 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:48:18.171864 17360 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:18.171998 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.251581 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:18.262717 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:18.349826 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:18.350090 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.393905 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.404605 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.421213 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:18.445330 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:18.446544 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:18.455469 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:18.480280 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240502 20:48:18.480988 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:18.491466 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.655802 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:18.666371 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:18.686772 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:48:18.687700 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.718036 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.755244 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:18.758682 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.771608 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 20:48:18.773958 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.791183 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:48:18.800673 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:18.887542 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:48:18.925999 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(6a8ce1df86914b2caf2d083d97eda4e6): perf score=0.033386
I20240502 20:48:18.941248 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(6a8ce1df86914b2caf2d083d97eda4e6) complete. Timing: real 0.015s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":261,"dirs.run_cpu_time_us":335,"dirs.run_wall_time_us":1562,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":531,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":202,"threads_started":2,"wal-append.queue_time_us":214}
I20240502 20:48:18.942405 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(843a26fc48844bfd82911b6a27833023): perf score=0.033389
I20240502 20:48:18.949982 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(f00c4b7954564820875a25da5834416a): perf score=0.033396
W20240502 20:48:18.951598 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:18.957406 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(843a26fc48844bfd82911b6a27833023) complete. Timing: real 0.015s	user 0.014s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":256,"dirs.run_cpu_time_us":333,"dirs.run_wall_time_us":1476,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":876,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":190,"threads_started":1,"wal-append.queue_time_us":333}
I20240502 20:48:18.958036 17614 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:18.958159 17614 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
W20240502 20:48:18.958740 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:18.958849 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(6a8ce1df86914b2caf2d083d97eda4e6): 4840 bytes on disk
I20240502 20:48:18.959854 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:48:18.959962 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:18.960049 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:48:18.960072 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:18.960126 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:48:18.958721 17614 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:48:18.960289 17033 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:48:18.961006 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(6a8ce1df86914b2caf2d083d97eda4e6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 20:48:18.962306 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(843a26fc48844bfd82911b6a27833023): 4586 bytes on disk
I20240502 20:48:18.962864 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(843a26fc48844bfd82911b6a27833023) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 20:48:18.967950 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(f00c4b7954564820875a25da5834416a) complete. Timing: real 0.018s	user 0.013s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":274,"dirs.run_cpu_time_us":417,"dirs.run_wall_time_us":1739,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":620,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":191,"threads_started":2,"wal-append.queue_time_us":206}
I20240502 20:48:18.969398 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(f00c4b7954564820875a25da5834416a): 4925 bytes on disk
I20240502 20:48:18.969907 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(f00c4b7954564820875a25da5834416a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 20:48:18.970875 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(f00c4b7954564820875a25da5834416a): perf score=0.033399
I20240502 20:48:18.976449 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(843a26fc48844bfd82911b6a27833023): perf score=0.033399
W20240502 20:48:18.988569 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:48:18.989626 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(f00c4b7954564820875a25da5834416a) complete. Timing: real 0.019s	user 0.018s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":397,"dirs.run_cpu_time_us":320,"dirs.run_wall_time_us":1443,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":590,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":296,"threads_started":2,"wal-append.queue_time_us":251}
I20240502 20:48:18.990931 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(f00c4b7954564820875a25da5834416a): 4925 bytes on disk
I20240502 20:48:18.991681 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(f00c4b7954564820875a25da5834416a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":213,"lbm_reads_lt_1ms":4}
I20240502 20:48:18.992585 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(6a8ce1df86914b2caf2d083d97eda4e6): perf score=0.033403
I20240502 20:48:18.994457 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(843a26fc48844bfd82911b6a27833023) complete. Timing: real 0.018s	user 0.013s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":313,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":1464,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":640,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":270,"threads_started":2,"wal-append.queue_time_us":220}
I20240502 20:48:18.995417 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(843a26fc48844bfd82911b6a27833023): 4586 bytes on disk
I20240502 20:48:18.995882 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(843a26fc48844bfd82911b6a27833023) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240502 20:48:19.007828 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:19.008458 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(6a8ce1df86914b2caf2d083d97eda4e6) complete. Timing: real 0.016s	user 0.014s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":105,"dirs.run_cpu_time_us":303,"dirs.run_wall_time_us":1600,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":565,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":139,"threads_started":1,"wal-append.queue_time_us":271}
I20240502 20:48:19.009608 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(6a8ce1df86914b2caf2d083d97eda4e6): 4840 bytes on disk
I20240502 20:48:19.010107 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(6a8ce1df86914b2caf2d083d97eda4e6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240502 20:48:19.052244 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:19.057011 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:19.131620 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:19.133680 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:19.185456 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:19.233765 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:48:19.284822 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:19.308430 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:19.332528 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:19.385673 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:19.466195 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(a2b61909194d488682afd91c0f796405): perf score=0.033387
I20240502 20:48:19.473127 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(62c71a33fbcc4cfdbf0065a3fc9dfeba): perf score=0.033392
I20240502 20:48:19.484545 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(a2b61909194d488682afd91c0f796405) complete. Timing: real 0.018s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":890,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":2308,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":700,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":190,"threads_started":2,"wal-append.queue_time_us":210}
W20240502 20:48:19.485127 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:48:19.485774 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(5586b9e60b544aed87c24f7b3ce0198c): perf score=0.033390
I20240502 20:48:19.491536 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(62c71a33fbcc4cfdbf0065a3fc9dfeba) complete. Timing: real 0.018s	user 0.015s	sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":214,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1286,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":682,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":206,"threads_started":2,"wal-append.queue_time_us":180}
I20240502 20:48:19.492642 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(5586b9e60b544aed87c24f7b3ce0198c): perf score=0.033394
I20240502 20:48:19.498750 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(62c71a33fbcc4cfdbf0065a3fc9dfeba): perf score=0.033395
I20240502 20:48:19.504201 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(5586b9e60b544aed87c24f7b3ce0198c) complete. Timing: real 0.018s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":292,"dirs.run_wall_time_us":1557,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":149,"threads_started":1,"wal-append.queue_time_us":481}
I20240502 20:48:19.506141 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(5586b9e60b544aed87c24f7b3ce0198c): 4840 bytes on disk
I20240502 20:48:19.506682 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(5586b9e60b544aed87c24f7b3ce0198c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240502 20:48:19.507668 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(a2b61909194d488682afd91c0f796405): 4586 bytes on disk
I20240502 20:48:19.508047 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(a2b61909194d488682afd91c0f796405) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 20:48:19.509593 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(5586b9e60b544aed87c24f7b3ce0198c) complete. Timing: real 0.017s	user 0.014s	sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":241,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":1455,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":105,"threads_started":1,"wal-append.queue_time_us":224}
I20240502 20:48:19.510568 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(62c71a33fbcc4cfdbf0065a3fc9dfeba): 4925 bytes on disk
I20240502 20:48:19.510974 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(62c71a33fbcc4cfdbf0065a3fc9dfeba) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 20:48:19.511766 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(5586b9e60b544aed87c24f7b3ce0198c): 4840 bytes on disk
I20240502 20:48:19.512156 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(5586b9e60b544aed87c24f7b3ce0198c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 20:48:19.513303 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(62c71a33fbcc4cfdbf0065a3fc9dfeba): perf score=0.033403
I20240502 20:48:19.516223 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(62c71a33fbcc4cfdbf0065a3fc9dfeba) complete. Timing: real 0.017s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":222,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1657,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":527,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":201,"threads_started":2,"wal-append.queue_time_us":202}
I20240502 20:48:19.517545 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(62c71a33fbcc4cfdbf0065a3fc9dfeba): 4925 bytes on disk
I20240502 20:48:19.518391 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(62c71a33fbcc4cfdbf0065a3fc9dfeba) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240502 20:48:19.529336 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(62c71a33fbcc4cfdbf0065a3fc9dfeba) complete. Timing: real 0.016s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":329,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":1399,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":483,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":285,"threads_started":2,"wal-append.queue_time_us":318}
I20240502 20:48:19.530709 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(a2b61909194d488682afd91c0f796405): perf score=0.033404
I20240502 20:48:19.544415 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(a2b61909194d488682afd91c0f796405) complete. Timing: real 0.013s	user 0.011s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1488,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":493,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":180,"threads_started":1,"wal-append.queue_time_us":385}
I20240502 20:48:19.545791 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(62c71a33fbcc4cfdbf0065a3fc9dfeba): 4925 bytes on disk
I20240502 20:48:19.546491 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(62c71a33fbcc4cfdbf0065a3fc9dfeba) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 20:48:19.547485 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(a2b61909194d488682afd91c0f796405): 4586 bytes on disk
I20240502 20:48:19.548009 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(a2b61909194d488682afd91c0f796405) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240502 20:48:19.560475 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:19.568883 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:19.589166 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:19.612784 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:19.627009 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:19.639525 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:19.702605 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:19.802430 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:19.864476 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:19.885195 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:19.949978 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:19.986726 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:20.006489 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:20.033727 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:20.033756 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:20.090857 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:20.111270 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.127014 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:20.179637 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:20.261904 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(14a7bead40fb4eb0bec6b00804b52595): perf score=0.033384
I20240502 20:48:20.272359 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(84484d65e2bb4e23ac621a39434ce382): perf score=0.033389
I20240502 20:48:20.278581 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(14a7bead40fb4eb0bec6b00804b52595) complete. Timing: real 0.016s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":274,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1429,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":226,"threads_started":2,"wal-append.queue_time_us":222}
I20240502 20:48:20.279800 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(8deaa1f751034c15acd6ccaad37536c6): perf score=0.033387
W20240502 20:48:20.286998 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:20.290788 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(84484d65e2bb4e23ac621a39434ce382) complete. Timing: real 0.018s	user 0.012s	sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":249,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1276,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":895,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":217,"threads_started":2,"wal-append.queue_time_us":207}
I20240502 20:48:20.291875 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(84484d65e2bb4e23ac621a39434ce382): 4925 bytes on disk
I20240502 20:48:20.292306 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(84484d65e2bb4e23ac621a39434ce382) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240502 20:48:20.294754 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:20.295864 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(8deaa1f751034c15acd6ccaad37536c6) complete. Timing: real 0.016s	user 0.014s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":92,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1200,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":108,"threads_started":1,"wal-append.queue_time_us":200}
I20240502 20:48:20.297581 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(8deaa1f751034c15acd6ccaad37536c6): 4840 bytes on disk
I20240502 20:48:20.298079 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(8deaa1f751034c15acd6ccaad37536c6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240502 20:48:20.298957 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(14a7bead40fb4eb0bec6b00804b52595): 4586 bytes on disk
I20240502 20:48:20.299463 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(14a7bead40fb4eb0bec6b00804b52595) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240502 20:48:20.301736 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(14a7bead40fb4eb0bec6b00804b52595): perf score=0.033389
I20240502 20:48:20.315353 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(14a7bead40fb4eb0bec6b00804b52595) complete. Timing: real 0.013s	user 0.011s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":253,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":1475,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":219,"threads_started":2,"wal-append.queue_time_us":246}
I20240502 20:48:20.316533 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(14a7bead40fb4eb0bec6b00804b52595): 4586 bytes on disk
I20240502 20:48:20.317286 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(14a7bead40fb4eb0bec6b00804b52595) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240502 20:48:20.327594 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:20.350728 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.361625 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:20.420226 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.460446 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:20.488421 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:20.519917 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:20.537246 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:20.561009 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:20.567130 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.568894 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.577329 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:20.605276 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.658537 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:20.678582 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.747157 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.837335 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.838754 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:20.873028 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.890991 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:20.894348 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:20.904850 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:20.932579 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.943058 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:20.954635 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:20.992130 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:20.998448 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:21.019383 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(fdd2311ec0b645f5a79098a32c6deba2): perf score=0.033381
I20240502 20:48:21.037123 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(fdd2311ec0b645f5a79098a32c6deba2) complete. Timing: real 0.017s	user 0.016s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":207,"dirs.run_cpu_time_us":316,"dirs.run_wall_time_us":1782,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":184,"threads_started":2,"wal-append.queue_time_us":299}
I20240502 20:48:21.038437 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(fdd2311ec0b645f5a79098a32c6deba2): 4840 bytes on disk
I20240502 20:48:21.039041 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(fdd2311ec0b645f5a79098a32c6deba2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 20:48:21.046499 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(a906569ab3b14520b4876bc8d5450f7d): perf score=0.033390
I20240502 20:48:21.054179 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(bf189c19164442e5af0827f1fb7dc085): perf score=0.033392
W20240502 20:48:21.061786 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:48:21.071130 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(a906569ab3b14520b4876bc8d5450f7d) complete. Timing: real 0.024s	user 0.012s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":238,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":1535,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1348,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":210,"threads_started":2,"wal-append.queue_time_us":195}
I20240502 20:48:21.071436 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(fdd2311ec0b645f5a79098a32c6deba2): perf score=0.033398
I20240502 20:48:21.072237 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(bf189c19164442e5af0827f1fb7dc085): perf score=0.033396
I20240502 20:48:21.088167 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(bf189c19164442e5af0827f1fb7dc085) complete. Timing: real 0.034s	user 0.017s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":807,"dirs.run_cpu_time_us":369,"dirs.run_wall_time_us":1573,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":241,"threads_started":2,"wal-append.queue_time_us":214}
I20240502 20:48:21.089447 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(a906569ab3b14520b4876bc8d5450f7d): perf score=0.033400
W20240502 20:48:21.091879 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:21.092684 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(fdd2311ec0b645f5a79098a32c6deba2) complete. Timing: real 0.021s	user 0.013s	sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":576,"dirs.run_cpu_time_us":320,"dirs.run_wall_time_us":1659,"drs_written":1,"lbm_read_time_us":168,"lbm_reads_lt_1ms":4,"lbm_write_time_us":710,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":262,"threads_started":2,"wal-append.queue_time_us":213}
I20240502 20:48:21.093632 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(fdd2311ec0b645f5a79098a32c6deba2): 4840 bytes on disk
I20240502 20:48:21.094046 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(fdd2311ec0b645f5a79098a32c6deba2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 20:48:21.099588 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(bf189c19164442e5af0827f1fb7dc085) complete. Timing: real 0.027s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":1322,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":167,"threads_started":1,"wal-append.queue_time_us":265}
I20240502 20:48:21.100708 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(bf189c19164442e5af0827f1fb7dc085): 4925 bytes on disk
I20240502 20:48:21.101231 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(bf189c19164442e5af0827f1fb7dc085) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 20:48:21.102619 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(a906569ab3b14520b4876bc8d5450f7d): 4586 bytes on disk
I20240502 20:48:21.103719 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(a906569ab3b14520b4876bc8d5450f7d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240502 20:48:21.103976 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:48:21.106400 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(a906569ab3b14520b4876bc8d5450f7d) complete. Timing: real 0.017s	user 0.014s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":88,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1512,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":682,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":82,"threads_started":1,"wal-append.queue_time_us":222}
I20240502 20:48:21.107576 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(bf189c19164442e5af0827f1fb7dc085): 4925 bytes on disk
I20240502 20:48:21.108120 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(bf189c19164442e5af0827f1fb7dc085) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 20:48:21.108914 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(a906569ab3b14520b4876bc8d5450f7d): 4586 bytes on disk
I20240502 20:48:21.109556 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(a906569ab3b14520b4876bc8d5450f7d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240502 20:48:21.162937 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:21.245891 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:21.301203 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:21.368157 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:21.368599 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:21.393688 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:21.416517 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:21.435590 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:21.508153 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:21.542282 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(5704bdfe6c02437c8e9c8c3deaadb9e3): perf score=0.033345
I20240502 20:48:21.559093 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(5704bdfe6c02437c8e9c8c3deaadb9e3) complete. Timing: real 0.016s	user 0.010s	sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":368,"dirs.run_cpu_time_us":429,"dirs.run_wall_time_us":1967,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":736,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":264,"threads_started":2,"wal-append.queue_time_us":304}
I20240502 20:48:21.560268 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(ff9fef57f6eb490da536170a02b96212): perf score=0.033349
W20240502 20:48:21.566488 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:21.575554 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:21.578730 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(ff9fef57f6eb490da536170a02b96212) complete. Timing: real 0.018s	user 0.016s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":70,"dirs.run_cpu_time_us":367,"dirs.run_wall_time_us":1491,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":738,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":89,"threads_started":1,"wal-append.queue_time_us":235}
I20240502 20:48:21.580101 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(ff9fef57f6eb490da536170a02b96212): 4925 bytes on disk
I20240502 20:48:21.581159 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(ff9fef57f6eb490da536170a02b96212) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240502 20:48:21.582584 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:21.582635 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(5704bdfe6c02437c8e9c8c3deaadb9e3): 4840 bytes on disk
I20240502 20:48:21.583068 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(5704bdfe6c02437c8e9c8c3deaadb9e3) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240502 20:48:21.589644 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:48:21.598918 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:21.606400 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(b5f81e46ab1f48fc8a88495b42f83640): perf score=0.033363
I20240502 20:48:21.613008 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(ff9fef57f6eb490da536170a02b96212): perf score=0.033359
I20240502 20:48:21.624614 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(b5f81e46ab1f48fc8a88495b42f83640) complete. Timing: real 0.018s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":808,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":1979,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":605,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":235,"threads_started":2,"wal-append.queue_time_us":235}
I20240502 20:48:21.626060 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(b5f81e46ab1f48fc8a88495b42f83640): 4586 bytes on disk
I20240502 20:48:21.626655 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(b5f81e46ab1f48fc8a88495b42f83640) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 20:48:21.636497 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(ff9fef57f6eb490da536170a02b96212) complete. Timing: real 0.023s	user 0.018s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":247,"dirs.run_cpu_time_us":421,"dirs.run_wall_time_us":1538,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":726,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":186,"threads_started":2,"wal-append.queue_time_us":222}
I20240502 20:48:21.637979 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(ff9fef57f6eb490da536170a02b96212): 4925 bytes on disk
I20240502 20:48:21.638749 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(ff9fef57f6eb490da536170a02b96212) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240502 20:48:21.642203 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:21.720432 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:21.786962 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:21.811092 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:21.854789 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:21.896392 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:21.916954 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:21.964303 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:21.986204 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.017591 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:22.045311 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.094122 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:22.110865 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.206017 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:22.291069 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:22.304164 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.314699 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.371845 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:22.380707 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(a547561343304137a044ce11517f6007): perf score=0.033335
W20240502 20:48:22.385145 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:48:22.393405 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(cd1b1316da2d47938164ac5b75f11bae): perf score=0.033338
I20240502 20:48:22.398818 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(a547561343304137a044ce11517f6007) complete. Timing: real 0.018s	user 0.012s	sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":234,"dirs.run_cpu_time_us":296,"dirs.run_wall_time_us":1742,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":871,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":265,"threads_started":2,"wal-append.queue_time_us":275}
I20240502 20:48:22.399952 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(cd1b1316da2d47938164ac5b75f11bae): perf score=0.033338
I20240502 20:48:22.409057 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(cd1b1316da2d47938164ac5b75f11bae) complete. Timing: real 0.015s	user 0.013s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":254,"dirs.run_cpu_time_us":301,"dirs.run_wall_time_us":1566,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":704,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":225,"threads_started":2,"wal-append.queue_time_us":213}
I20240502 20:48:22.410274 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(cd1b1316da2d47938164ac5b75f11bae): 4586 bytes on disk
I20240502 20:48:22.410831 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(cd1b1316da2d47938164ac5b75f11bae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 20:48:22.416282 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(cd1b1316da2d47938164ac5b75f11bae) complete. Timing: real 0.016s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1904,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":850,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":89,"threads_started":1,"wal-append.queue_time_us":278}
I20240502 20:48:22.417469 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(a547561343304137a044ce11517f6007): 4840 bytes on disk
I20240502 20:48:22.417887 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(a547561343304137a044ce11517f6007) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 20:48:22.418757 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(cd1b1316da2d47938164ac5b75f11bae): 4586 bytes on disk
I20240502 20:48:22.419222 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(cd1b1316da2d47938164ac5b75f11bae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240502 20:48:22.451287 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.451858 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.469025 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:22.498024 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 20:48:22.588255 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(135a08f1f2ac4707bbacee81ac73a19e): perf score=0.033395
I20240502 20:48:22.600991 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(cd1b1316da2d47938164ac5b75f11bae): perf score=0.033393
I20240502 20:48:22.601980 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(135a08f1f2ac4707bbacee81ac73a19e) complete. Timing: real 0.013s	user 0.010s	sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":248,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1778,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":639,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":176,"threads_started":2,"wal-append.queue_time_us":211}
I20240502 20:48:22.603010 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(135a08f1f2ac4707bbacee81ac73a19e): 4925 bytes on disk
I20240502 20:48:22.603459 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(135a08f1f2ac4707bbacee81ac73a19e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 20:48:22.604697 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(a547561343304137a044ce11517f6007): perf score=0.033398
W20240502 20:48:22.614167 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:22.615406 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(cd1b1316da2d47938164ac5b75f11bae) complete. Timing: real 0.014s	user 0.008s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":236,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":1480,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":544,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":181,"threads_started":2,"wal-append.queue_time_us":202}
W20240502 20:48:22.616003 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:22.616811 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(cd1b1316da2d47938164ac5b75f11bae): 4586 bytes on disk
I20240502 20:48:22.617404 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(cd1b1316da2d47938164ac5b75f11bae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 20:48:22.623610 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(a547561343304137a044ce11517f6007) complete. Timing: real 0.019s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1427,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1697,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":159,"threads_started":1,"wal-append.queue_time_us":289}
I20240502 20:48:22.624622 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(a547561343304137a044ce11517f6007): 4840 bytes on disk
I20240502 20:48:22.625412 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(a547561343304137a044ce11517f6007) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
W20240502 20:48:22.709376 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.771426 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:22.775604 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.797021 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.799943 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:22.818773 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:22.860895 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:22.865777 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:22.895237 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:22.906672 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:22.917790 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:22.985582 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:22.992844 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.011348 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.050441 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:23.073204 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:23.073204 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:23.151984 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:23.164578 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(40df15a82a624913b3b712687d999134): perf score=0.033338
I20240502 20:48:23.172807 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(f00d12e425284c36ae8baed56c7cfaf6): perf score=0.033343
I20240502 20:48:23.177816 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(40df15a82a624913b3b712687d999134) complete. Timing: real 0.013s	user 0.011s	sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":253,"dirs.run_cpu_time_us":361,"dirs.run_wall_time_us":1671,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":191}
I20240502 20:48:23.179044 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(40df15a82a624913b3b712687d999134): 4586 bytes on disk
I20240502 20:48:23.179545 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(40df15a82a624913b3b712687d999134) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240502 20:48:23.180873 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(f00d12e425284c36ae8baed56c7cfaf6): perf score=0.033340
I20240502 20:48:23.188004 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(f00d12e425284c36ae8baed56c7cfaf6) complete. Timing: real 0.015s	user 0.013s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":200,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1494,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":661,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":170}
I20240502 20:48:23.188884 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(62ebc6576a4e4a108de55a4e9d7ff6b8): perf score=0.033346
I20240502 20:48:23.194723 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(f00d12e425284c36ae8baed56c7cfaf6) complete. Timing: real 0.014s	user 0.012s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1368,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":518,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":150,"threads_started":1,"wal-append.queue_time_us":243}
I20240502 20:48:23.196117 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(f00d12e425284c36ae8baed56c7cfaf6): 4840 bytes on disk
I20240502 20:48:23.196697 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(f00d12e425284c36ae8baed56c7cfaf6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 20:48:23.204475 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(62ebc6576a4e4a108de55a4e9d7ff6b8) complete. Timing: real 0.015s	user 0.014s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":80,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1196,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":418,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":121,"threads_started":1,"wal-append.queue_time_us":215}
I20240502 20:48:23.205462 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(62ebc6576a4e4a108de55a4e9d7ff6b8): 4923 bytes on disk
I20240502 20:48:23.205865 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(62ebc6576a4e4a108de55a4e9d7ff6b8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 20:48:23.206552 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(f00d12e425284c36ae8baed56c7cfaf6): 4840 bytes on disk
I20240502 20:48:23.206959 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(f00d12e425284c36ae8baed56c7cfaf6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240502 20:48:23.211340 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:23.232254 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.277446 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:23.292034 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:23.301936 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.341953 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.345000 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:23.363899 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:23.368026 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:23.371647 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(40df15a82a624913b3b712687d999134): perf score=0.033400
I20240502 20:48:23.379091 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(62ebc6576a4e4a108de55a4e9d7ff6b8): perf score=0.033398
I20240502 20:48:23.386134 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(40df15a82a624913b3b712687d999134) complete. Timing: real 0.014s	user 0.013s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":194,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":1439,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":471,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":194,"threads_started":2,"wal-append.queue_time_us":217}
I20240502 20:48:23.387118 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(40df15a82a624913b3b712687d999134): 4586 bytes on disk
I20240502 20:48:23.387629 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(40df15a82a624913b3b712687d999134) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 20:48:23.395607 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(62ebc6576a4e4a108de55a4e9d7ff6b8) complete. Timing: real 0.016s	user 0.012s	sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":209,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1388,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":493,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":202,"threads_started":2,"wal-append.queue_time_us":231}
I20240502 20:48:23.397120 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(40df15a82a624913b3b712687d999134): perf score=0.033401
I20240502 20:48:23.408644 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(40df15a82a624913b3b712687d999134) complete. Timing: real 0.011s	user 0.010s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":82,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1153,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":132,"threads_started":1,"wal-append.queue_time_us":203}
I20240502 20:48:23.409554 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(62ebc6576a4e4a108de55a4e9d7ff6b8): 4923 bytes on disk
I20240502 20:48:23.409965 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(62ebc6576a4e4a108de55a4e9d7ff6b8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 20:48:23.411049 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(40df15a82a624913b3b712687d999134): 4586 bytes on disk
I20240502 20:48:23.411378 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(40df15a82a624913b3b712687d999134) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240502 20:48:23.434371 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.499558 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 178 similar messages]
W20240502 20:48:23.570868 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:23.587375 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.650933 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:23.691741 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 20:48:23.692528 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:23.692553 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.694631 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.716969 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:23.739185 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.784929 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240502 20:48:23.823320 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:23.823315 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:23.840458 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:23.890092 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(af693f5816e046f592fbd9b2e7ef0ad9): perf score=0.033351
W20240502 20:48:23.891793 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:48:23.904358 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(af693f5816e046f592fbd9b2e7ef0ad9) complete. Timing: real 0.014s	user 0.012s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":251,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":1387,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":179,"threads_started":2,"wal-append.queue_time_us":186}
I20240502 20:48:23.905512 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(af693f5816e046f592fbd9b2e7ef0ad9): 4840 bytes on disk
I20240502 20:48:23.906075 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(af693f5816e046f592fbd9b2e7ef0ad9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240502 20:48:23.913743 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(f1545142b0bd40859bf33bf8686002ae): perf score=0.033361
I20240502 20:48:23.929000 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(f1545142b0bd40859bf33bf8686002ae) complete. Timing: real 0.015s	user 0.012s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":274,"dirs.run_cpu_time_us":312,"dirs.run_wall_time_us":1591,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":528,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":225,"threads_started":2,"wal-append.queue_time_us":225}
I20240502 20:48:23.930495 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(f1545142b0bd40859bf33bf8686002ae): 4925 bytes on disk
I20240502 20:48:23.931299 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(f1545142b0bd40859bf33bf8686002ae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240502 20:48:23.932528 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(af693f5816e046f592fbd9b2e7ef0ad9): perf score=0.033363
I20240502 20:48:23.947489 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(af693f5816e046f592fbd9b2e7ef0ad9) complete. Timing: real 0.014s	user 0.012s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":1555,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":682,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":95,"threads_started":1,"wal-append.queue_time_us":247}
I20240502 20:48:23.948693 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(af693f5816e046f592fbd9b2e7ef0ad9): 4840 bytes on disk
I20240502 20:48:23.949230 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(af693f5816e046f592fbd9b2e7ef0ad9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
I20240502 20:48:23.950790 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(5cf737e20ca74db1b94ccff5d60133ee): perf score=0.033371
W20240502 20:48:23.956291 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:23.956411 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:48:23.960331 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(f1545142b0bd40859bf33bf8686002ae): perf score=0.033372
I20240502 20:48:23.965426 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(5cf737e20ca74db1b94ccff5d60133ee) complete. Timing: real 0.014s	user 0.014s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3805,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":249,"threads_started":2,"wal-append.queue_time_us":186}
I20240502 20:48:23.966722 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(5cf737e20ca74db1b94ccff5d60133ee): 4586 bytes on disk
I20240502 20:48:23.967289 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(5cf737e20ca74db1b94ccff5d60133ee) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 20:48:23.974511 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(f1545142b0bd40859bf33bf8686002ae) complete. Timing: real 0.014s	user 0.012s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":237,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":1455,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":567,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":217,"threads_started":2,"wal-append.queue_time_us":196}
I20240502 20:48:23.975394 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(f1545142b0bd40859bf33bf8686002ae): 4925 bytes on disk
I20240502 20:48:23.975812 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(f1545142b0bd40859bf33bf8686002ae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 20:48:23.976478 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling FlushMRSOp(5cf737e20ca74db1b94ccff5d60133ee): perf score=0.033375
I20240502 20:48:23.989619 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: FlushMRSOp(5cf737e20ca74db1b94ccff5d60133ee) complete. Timing: real 0.013s	user 0.011s	sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1402,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":481,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":104,"threads_started":1,"wal-append.queue_time_us":209}
I20240502 20:48:23.990464 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(5cf737e20ca74db1b94ccff5d60133ee): 4586 bytes on disk
I20240502 20:48:23.990870 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(5cf737e20ca74db1b94ccff5d60133ee) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240502 20:48:24.105494 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:24.157774 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:24.178828 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:24.238034 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:24.239554 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:24.258199 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:48:24.282678 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:48:24.309528 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:24.346479 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:24.431476 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:24.445812 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:24.482620 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:24.495294 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:24.574075 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:24.601714 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:24.654959 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:24.671377 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:24.690891 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:24.703414 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:24.740526 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:24.747275 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:24.764828 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:24.801877 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:24.839222 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:24.953505 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:24.998034 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.038125 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.050892 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.080374 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.156296 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:25.179028 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:25.190222 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.259764 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.319584 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:25.353331 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:25.404228 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.404915 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:25.428212 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.448992 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.522046 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.525585 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.530835 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.549122 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.615244 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:25.615912 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.639799 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.655306 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.689967 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.710346 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:25.711063 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.747196 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:25.753718 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.754416 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.792095 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.862295 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.919677 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:25.932021 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:25.942154 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:25.946410 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:25.980798 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:26.081379 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:26.162137 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:26.177901 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:26.181622 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:26.193521 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 20:48:26.209743 18378 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:48:26.210026 18378 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:48:26.210981 18378 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:48:26.211441 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:48:26.211702 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:48:26.212287 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:26.212491 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:48:26.212666 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:26.212771 18378 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:48:26.264535 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:26.297457 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:26.345963 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:26.431409 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:26.454294 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:26.501792 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:26.560088 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:26.663367 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:48:26.679419 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:26.742087 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:26.769666 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:26.780879 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:26.787441 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:26.834559 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:26.903972 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:26.912896 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:26.990283 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:26.993414 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:26.999564 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.066895 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.067387 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:27.108598 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.119630 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:27.172046 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:27.230226 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:27.236069 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.256086 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.269483 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:27.311630 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.351248 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:27.360332 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.441751 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:27.521361 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.578851 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:27.615103 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:27.629900 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.735567 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:27.768821 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:27.794679 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.805491 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.859287 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:27.875938 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:27.966941 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:27.989317 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:27.996766 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:28.013037 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:28.027652 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:28.035172 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:28.063758 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.116498 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:28.127434 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.168614 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.217562 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:28.226374 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.231590 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:28.237150 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:28.261983 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.265264 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:28.312646 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:28.332147 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:28.338886 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:28.387153 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:28.429471 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.431927 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.510109 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.525578 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:28.562774 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 20:48:28.583842 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.586337 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.661384 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.689517 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:28.729321 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:48:28.730163 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:28.747507 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.778486 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.793701 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 20:48:28.794608 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:28.805261 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:28.836217 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:48:29.011780 19248 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:48:29.011981 19248 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:48:29.012534 19248 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:48:29.014175 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:29.014489 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:48:29.014617 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:48:29.014786 19248 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:29.077836 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:29.080195 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:29.101169 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:29.141920 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:29.188777 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:29.236342 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:29.253473 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:29.256933 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:29.266590 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:29.340159 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:29.351131 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:48:29.353902 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:48:29.365875 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:29.475801 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:29.479074 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:29.488922 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:29.577792 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:29.624130 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:29.674713 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:29.679896 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:29.692286 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:29.718477 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:29.733323 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:48:29.750393 19411 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:29.750576 19411 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:48:29.751111 19411 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:48:29.751812 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:48:29.751984 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:48:29.752875 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:29.753206 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:29.753299 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:48:29.753460 19411 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:29.791728 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:29.798372 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:29.808540 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:29.841714 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:29.923735 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:29.951953 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:30.009529 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:30.072810 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:30.075376 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:30.082454 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:30.173732 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:30.239589 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:30.253651 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:30.344641 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:30.347558 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:30.356312 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:48:30.364526 19072 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:30.364763 19072 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:48:30.365599 19072 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:48:30.366165 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:48:30.366400 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:48:30.367312 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:30.367524 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:30.367636 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:48:30.367941 19072 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:30.390667 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:30.477129 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:30.522310 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:30.553890 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:30.589970 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:30.629630 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:30.631982 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:30.671756 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:30.679736 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:30.704391 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:30.712042 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:30.717429 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:30.744686 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:30.788831 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:30.800220 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:30.806869 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:30.832293 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:30.856166 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:30.857738 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:30.934666 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:30.998605 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:31.037638 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:31.040426 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:31.055217 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:31.085556 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:31.097354 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:31.097363 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:31.145431 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:31.164320 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:31.209178 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:31.255329 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:31.318379 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:31.357014 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:31.555450 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:31.629545 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:31.679510 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:31.696141 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:31.700637 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:31.739666 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:31.758103 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:31.847424 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:31.966848 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:32.005824 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:48:32.059818 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:32.060039 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.065928 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:32.072038 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.101879 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.135087 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.145890 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.181617 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:32.189216 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:32.193568 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.196089 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.210074 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:32.309422 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.310266 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:32.370692 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:32.403590 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.407120 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.461560 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:32.488126 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.511497 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.531562 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:32.557040 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.631093 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:32.658247 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:48:32.690893 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(ff9fef57f6eb490da536170a02b96212): perf score=0.033394
I20240502 20:48:32.707512 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(ff9fef57f6eb490da536170a02b96212) complete. Timing: real 0.016s	user 0.014s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":266,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":2017,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":775,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":177,"threads_started":2,"wal-append.queue_time_us":159}
I20240502 20:48:32.708549 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(ff9fef57f6eb490da536170a02b96212): 4925 bytes on disk
I20240502 20:48:32.709045 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(ff9fef57f6eb490da536170a02b96212) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240502 20:48:32.758611 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.822104 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:32.877700 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.896148 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.921244 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:32.949489 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.029721 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:33.069967 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:33.083573 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.109124 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.115746 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:33.123710 19072 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:33.124020 19072 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:48:33.124979 19072 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:48:33.126065 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:48:33.126310 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:48:33.126767 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:33.126946 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:33.127063 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:48:33.127300 19072 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:33.137091 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:33.204602 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.254212 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:33.258611 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.270442 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.306258 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.310633 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:33.365706 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:33.404714 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.411011 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.414451 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:33.417837 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.447851 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:48:33.491500 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling FlushMRSOp(84484d65e2bb4e23ac621a39434ce382): perf score=0.033353
I20240502 20:48:33.506649 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: FlushMRSOp(84484d65e2bb4e23ac621a39434ce382) complete. Timing: real 0.015s	user 0.011s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":396,"dirs.run_cpu_time_us":340,"dirs.run_wall_time_us":2014,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":604,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":896,"thread_start_us":250,"threads_started":2,"wal-append.queue_time_us":218}
I20240502 20:48:33.508066 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(84484d65e2bb4e23ac621a39434ce382): 4925 bytes on disk
I20240502 20:48:33.508613 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(84484d65e2bb4e23ac621a39434ce382) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240502 20:48:33.513608 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:33.549206 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.556509 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:33.564855 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.589217 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 184 similar messages]
W20240502 20:48:33.590274 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:33.602692 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.637452 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:33.675293 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:33.733006 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 20:48:33.734202 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:33.738785 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:33.760991 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:33.815955 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 20:48:33.853808 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:48:33.885617 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:34.032308 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:48:34.098600 19914 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:34.098786 19914 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:48:34.099437 19914 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:48:34.099879 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:48:34.100045 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:48:34.100867 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:34.101015 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:34.101058 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:48:34.101290 19914 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:34.103477 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:34.166875 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:34.203233 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:48:34.215977 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(b5f81e46ab1f48fc8a88495b42f83640): perf score=0.033387
I20240502 20:48:34.231163 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(b5f81e46ab1f48fc8a88495b42f83640) complete. Timing: real 0.015s	user 0.010s	sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":262,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":1641,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":300,"threads_started":2,"wal-append.queue_time_us":277}
I20240502 20:48:34.232196 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(b5f81e46ab1f48fc8a88495b42f83640): 4586 bytes on disk
I20240502 20:48:34.232651 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(b5f81e46ab1f48fc8a88495b42f83640) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240502 20:48:34.233377 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling FlushMRSOp(d1b9ff3c921d474fa5ad106d1c17ab00): perf score=0.033377
W20240502 20:48:34.236997 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:34.245797 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:48:34.247771 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: FlushMRSOp(d1b9ff3c921d474fa5ad106d1c17ab00) complete. Timing: real 0.014s	user 0.012s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":143,"dirs.run_cpu_time_us":337,"dirs.run_wall_time_us":1527,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":544,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136,"threads_started":1,"wal-append.queue_time_us":258}
I20240502 20:48:34.248733 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(d1b9ff3c921d474fa5ad106d1c17ab00): 4925 bytes on disk
I20240502 20:48:34.249538 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(d1b9ff3c921d474fa5ad106d1c17ab00) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 20:48:34.260753 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling FlushMRSOp(f2dfc0bbbbca4cc28a97480007343424): perf score=0.033376
W20240502 20:48:34.266619 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:48:34.274502 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: FlushMRSOp(f2dfc0bbbbca4cc28a97480007343424) complete. Timing: real 0.013s	user 0.012s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":323,"dirs.run_cpu_time_us":331,"dirs.run_wall_time_us":1625,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":610,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":220,"threads_started":2,"wal-append.queue_time_us":294}
I20240502 20:48:34.275509 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(f2dfc0bbbbca4cc28a97480007343424): 4925 bytes on disk
I20240502 20:48:34.275943 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(f2dfc0bbbbca4cc28a97480007343424) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240502 20:48:34.284265 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:34.336933 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:34.346027 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:34.410809 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 20:48:34.499382 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:34.536315 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:34.596135 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:34.626287 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:34.641520 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:34.658048 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:34.679044 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:34.687920 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:34.704357 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:34.756004 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:34.766115 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:34.785593 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:34.876468 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:34.881274 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:34.912067 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:34.977412 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:34.997162 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:35.039099 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:35.059664 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:35.068584 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:35.098640 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:35.147717 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:35.226516 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:35.241802 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:35.326350 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:35.341651 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:35.378934 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:35.414578 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:35.453364 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:35.459231 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:35.497828 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:35.543939 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:35.609426 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:35.634881 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:35.666514 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:35.669355 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:35.677908 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:35.725174 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:35.762477 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:35.836737 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:35.861822 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:35.880796 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:35.883183 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:35.884243 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:35.894675 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:35.938688 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:35.948575 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:36.045120 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:36.052191 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:36.062256 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:36.067358 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:36.068826 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:36.073501 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:36.125229 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:36.139745 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:36.176589 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:36.215065 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:36.344516 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:36.347344 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:36.368752 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:36.472322 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:36.536253 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:36.583281 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:36.634812 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:36.758144 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:36.761827 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:36.764578 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:36.779196 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:48:36.842908 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:36.925374 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.025538 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:37.039942 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.056231 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.076102 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.080580 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:37.093955 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.126497 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.129874 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:37.164614 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:37.215457 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:37.243664 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:37.263084 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.441015 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.441327 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:37.484583 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.490298 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:37.560046 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:37.567327 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:37.590253 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:37.644603 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.649493 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.665131 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.706378 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:37.812359 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:37.849200 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.880689 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.889669 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:37.906316 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:37.908031 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:37.955415 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:38.076251 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:38.104326 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:38.122026 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.142609 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:38.144028 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:38.179194 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:38.223608 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.259909 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:38.281391 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.290827 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:38.313241 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.344208 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.469563 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.474421 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.484768 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:38.495435 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.535030 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:38.543145 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:38.566579 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:38.587759 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.610337 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.650969 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 178 similar messages]
W20240502 20:48:38.682988 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:38.705188 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:38.709769 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:38.729753 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.768806 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:38.781704 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:38.787704 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240502 20:48:38.803896 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.821892 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 20:48:38.822930 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.920137 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:38.945209 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:38.992146 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:39.107137 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:39.137339 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:39.179802 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:39.248339 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:39.312862 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:39.313026 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:39.318325 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:39.361984 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:48:39.412077 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:39.474092 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:48:39.475026 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:39.493872 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:39.529578 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:39.553114 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:39.568723 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:39.576929 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:39.579450 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:39.623585 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:39.630443 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:39.632068 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:39.647614 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:39.742585 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:39.759318 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:39.893201 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:39.921484 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:39.923059 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:39.938259 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:39.980682 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:40.054967 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:40.073475 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:40.241505 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:40.249749 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:40.258226 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:40.259855 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:40.310652 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:40.324281 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:40.343187 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:40.390935 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:40.444589 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:40.480517 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:40.484709 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:40.487075 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:40.532380 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:40.538543 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:40.561501 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:40.582247 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:40.589746 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:40.672394 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:40.853142 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:40.861272 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:40.865932 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:40.879359 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:40.895709 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:40.943898 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.003857 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:41.037626 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.052438 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.072957 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.075716 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:41.124346 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:41.130894 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.132800 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:41.161631 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 20:48:41.181818 21203 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:48:41.182013 21203 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:48:41.182926 21203 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:48:41.184015 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:48:41.184363 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:41.184485 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:41.184975 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:48:41.185102 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:48:41.185438 21203 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:48:41.192246 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.200575 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.221838 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.228345 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:41.289520 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.290040 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:41.293008 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:41.318747 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:41.351696 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:41.432719 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.524863 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:41.544880 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:41.604811 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:41.618825 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:41.697280 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.768740 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:41.798583 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:41.841416 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:48:41.908023 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:41.986371 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:41.987978 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.043623 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:42.111588 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.111600 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:42.111619 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.114701 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:42.122757 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:42.149132 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.257198 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:42.274119 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:42.355374 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:42.381002 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.406280 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:42.412185 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.465879 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.498277 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:42.513484 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.527769 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.587112 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:42.621335 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.713963 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.727335 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:42.729209 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.733479 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.770862 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:42.816087 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:42.907390 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:42.910823 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:42.997233 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:43.001308 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:43.036082 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:43.094018 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.104681 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.152710 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.178217 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.198681 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:43.201567 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:43.219640 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:43.236518 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:43.293385 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.403815 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:43.497936 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.497936 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.517105 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:43.527256 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.527374 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.614939 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:43.663200 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:48:43.678151 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:43.680716 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.691856 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:43.710497 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:43.711722 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:43.739141 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.758759 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.783203 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:43.836253 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 20:48:43.872002 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:43.935717 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:43.942117 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:43.949425 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:43.957281 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240502 20:48:43.958359 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:48:44.052016 21717 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:48:44.052237 21717 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:48:44.053004 21717 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:48:44.054188 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:44.054291 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:44.054335 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:48:44.054517 21717 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:44.074980 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.106269 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:44.125422 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:44.140170 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:44.163658 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.167551 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:48:44.187441 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:44.228564 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:44.251413 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.457504 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:44.466434 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:44.526257 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:48:44.530229 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:44.563112 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:44.569972 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:44.606792 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:44.631681 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.654862 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.685029 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:44.762293 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.795542 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.837024 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:44.899930 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.915841 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.922544 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.971439 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:44.983160 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.996165 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:44.997311 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:45.081153 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:45.085417 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:45.089221 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:45.095857 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:45.101562 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:45.135361 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:45.153506 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:45.161653 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:45.214730 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:45.316085 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:45.366695 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:45.374240 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:45.499684 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:45.512586 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:45.513957 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:45.519142 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:45.545598 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:45.561569 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:45.604653 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:45.616830 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:45.736057 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:45.766954 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:45.830407 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:45.871166 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:45.910053 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:45.938271 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:46.002732 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:46.020004 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:46.054476 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:46.071331 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:46.082084 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:46.084955 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:46.129360 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:46.174206 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:46.217453 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:46.233107 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:46.258374 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:46.272071 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:46.293995 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:46.425478 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:46.425570 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:46.488912 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:46.496891 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:48:46.513507 21441 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:46.513684 21441 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:48:46.514288 21441 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:48:46.514883 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:48:46.515071 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:48:46.515420 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:46.515600 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:46.515673 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:48:46.515833 21441 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:46.537045 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:46.587756 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:46.604604 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:46.611999 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:46.672673 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:46.758636 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:46.767089 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:48:46.773869 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:46.816504 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:46.821600 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:46.842303 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:48:47.009203 22051 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:47.009361 22051 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:48:47.009953 22051 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:48:47.010901 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:48:47.011085 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:48:47.011763 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:47.011981 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:47.012034 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:48:47.012230 22051 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:47.026018 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:47.056542 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:47.105831 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:47.208829 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:47.220957 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:47.221163 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.230680 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.237382 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:47.237599 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.294258 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.326471 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:47.335871 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.420469 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.440616 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.452642 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:47.498643 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.525234 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:47.530128 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.552011 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.564338 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.579411 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:47.639175 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:47.709153 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.709949 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:48:47.711413 22532 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:48:47.711567 22532 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:48:47.712179 22532 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:48:47.712718 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:48:47.713551 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:48:47.713830 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:47.714066 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:48:47.714218 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:47.714249 22532 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:48:47.721760 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.747507 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:47.770782 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:47.842182 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:47.862993 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:47.885375 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:47.980867 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:47.988571 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:47.996681 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:48.088635 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.130187 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:48.213533 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:48.252453 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:48.274271 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.292690 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:48.302280 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.336750 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:48.402469 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:48.406656 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.414880 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:48.436902 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:48.449384 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:48.469650 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.494746 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.600140 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.665604 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:48.682920 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 176 similar messages]
W20240502 20:48:48.683858 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:48.730731 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:48.760706 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.817606 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:48.826380 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.856105 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 20:48:48.857174 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:48.858144 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.879369 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:48.909261 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:48.940711 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:48.987689 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 151 similar messages]
I20240502 20:48:49.026930 22823 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:48:49.027134 22823 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:48:49.027835 22823 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:48:49.029681 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:49.029822 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:48:49.029871 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:48:49.030071 22823 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:49.039784 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:49.053189 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:49.085131 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:49.128813 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:49.195873 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:49.207734 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:49.337980 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:49.345355 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:49.372457 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:49.387672 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:48:49.481016 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:49.483253 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:49.503880 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:49.506593 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:49.579830 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:49.628089 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:49.712494 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240502 20:48:49.713794 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:49.758765 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:49.764540 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:49.771229 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:49.809432 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:49.834540 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:49.882753 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:49.884552 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:49.892710 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:49.898276 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:49.908749 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:49.929874 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:49.991139 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:50.012377 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.065897 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.075351 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:50.100780 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.109136 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.135782 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.152168 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:50.232424 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.297574 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:50.327626 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.467394 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.471472 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:50.472708 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:50.475944 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:50.475983 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:50.575947 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:50.643143 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.658243 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:50.666456 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.670266 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:50.688344 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:50.738691 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:50.759964 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:50.792544 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.795120 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:50.918699 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:50.922701 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:50.990638 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:51.003556 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:51.070160 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:51.131759 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:51.134143 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:51.152710 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:51.194761 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:51.276420 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:51.280992 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:51.317799 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:51.363422 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:51.395644 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:51.428038 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:51.487614 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:51.490010 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:51.581331 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:51.583933 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:51.605010 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:51.642395 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:51.813758 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:51.819518 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:51.862308 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:51.962011 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:51.995962 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.024506 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.043947 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.055013 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:52.055052 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:52.055640 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:48:52.071044 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:52.100819 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:52.198824 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:52.230230 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.259868 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.277199 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:52.302120 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.305088 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.336282 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:52.370146 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.378590 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:52.393440 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.403734 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:52.419946 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.423333 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.459771 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:52.460757 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.584141 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.672886 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:52.751025 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:52.764011 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.772843 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.773887 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:52.774516 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:52.810068 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.838476 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:52.938344 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:52.984463 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:53.008231 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:53.033841 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:53.041949 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:53.071574 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:53.189934 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:53.206202 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:53.253582 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:53.263849 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:53.281559 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:53.304692 23530 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:53.304939 23530 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:48:53.305653 23530 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:48:53.306376 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:48:53.306593 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:48:53.307047 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:53.307271 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:48:53.307361 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:48:53.307543 23530 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:53.316279 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:53.350988 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:53.416985 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:53.472211 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:53.473821 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:53.476797 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:53.484338 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:53.591418 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:53.633297 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:53.670778 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:53.677109 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:53.681025 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:53.688591 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:48:53.689558 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:53.774641 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:53.806834 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:53.815872 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:53.819401 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:53.857827 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 20:48:53.901683 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:54.015323 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.025838 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240502 20:48:54.077697 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:54.145519 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:54.193392 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.207152 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.212038 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:54.216718 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:54.340703 23672 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:48:54.340885 23672 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:48:54.341497 23672 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:48:54.342891 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:54.343050 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:54.343147 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:48:54.343327 23672 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:54.370818 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.394500 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:54.418179 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:54.443923 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:54.500043 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:54.514201 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.518690 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.654435 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.654510 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.689558 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:54.720403 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:54.721089 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:54.735201 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:54.739954 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:48:54.741811 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:54.779708 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:54.800412 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.803292 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.811338 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.825932 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.828784 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:54.842056 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:54.883545 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:48:54.884375 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:55.000178 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:55.058943 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:55.084957 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:55.106251 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:55.126950 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:55.168922 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:55.180980 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:55.243995 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:55.286720 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:55.299656 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:55.301187 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:55.367564 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:55.406394 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:55.418299 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:55.467584 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:55.492658 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:55.540426 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:55.561123 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:55.604493 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:55.628574 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:55.680784 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:55.732494 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:55.745329 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:55.923182 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:55.941367 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:55.951454 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 20:48:55.957805 23338 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:48:55.957981 23338 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:48:55.958616 23338 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:48:55.959658 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:48:55.959924 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:48:55.960861 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:55.961236 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:55.961304 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:48:55.961504 23338 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:56.009809 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:56.009822 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:56.070796 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:56.073563 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:56.077490 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:56.079700 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:56.084054 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:56.095574 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:56.126760 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:56.137687 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:56.195189 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:56.232015 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:56.370553 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:56.405215 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:56.415608 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:56.465162 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:56.524437 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:56.569471 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:56.637923 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:56.644785 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:56.657737 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:56.671092 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:56.707919 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:56.767274 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:56.813233 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:48:56.846143 23955 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:48:56.846355 23955 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:48:56.846980 23955 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:48:56.849035 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:56.849243 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:48:56.849361 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:48:56.849555 23955 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:48:56.911846 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:56.946476 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:56.983264 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:56.994942 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.095628 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:57.146785 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.160656 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:57.164155 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:57.170792 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.176482 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.199031 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:57.205497 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:57.210743 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.250777 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:57.262943 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.274219 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.309469 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:57.320849 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:48:57.340268 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:57.377653 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.397873 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.449992 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.540016 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.575737 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:57.658067 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.661922 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:57.668460 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:57.710657 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:57.716542 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:57.776317 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.807408 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:57.862527 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:57.947538 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:57.954457 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.971902 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:57.974388 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:58.059774 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:58.070070 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:58.097541 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:58.121482 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:58.125483 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:58.192968 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:58.225622 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:58.274574 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:58.312067 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:58.476096 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:58.483045 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:58.502959 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:58.505427 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:58.545580 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:58.652220 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:58.703142 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:58.737975 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:58.748824 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:58.756156 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:58.769753 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 20:48:58.770545 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:58.810628 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:58.814404 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:58.816826 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:58.820034 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:58.931388 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:58.957461 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240502 20:48:59.002480 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.002795 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:48:59.118942 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 20:48:59.129958 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.152912 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:59.154610 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:59.180902 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.185812 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:59.187126 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:59.292588 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:59.323788 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:59.344172 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.405896 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.493099 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.550733 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:48:59.566895 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:48:59.588043 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:48:59.593284 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:48:59.594421 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:48:59.692838 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:48:59.704488 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:59.803391 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.804035 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.813458 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:48:59.815758 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:48:59.823851 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.831758 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.853372 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:48:59.868432 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:48:59.876320 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.883687 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.927690 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:48:59.928570 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:48:59.961844 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:00.013203 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:00.030850 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:00.046717 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:49:00.075369 24659 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:00.075563 24659 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:49:00.076406 24659 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:49:00.076983 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:00.077186 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:00.077953 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:00.078061 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:00.078120 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:49:00.078274 24659 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:00.080752 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:00.111905 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:00.260236 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:00.263983 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:00.272615 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:00.287708 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:00.349951 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:00.406842 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:00.411407 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:00.437897 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:00.447059 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:00.460709 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:00.467551 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:00.484798 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:00.549793 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:00.622821 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:00.669749 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:00.685761 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:00.695843 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:00.866822 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:00.956514 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:00.960733 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:00.989349 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:01.049260 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:01.049561 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:01.132090 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:01.166317 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:01.186544 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:01.187731 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:01.223436 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:01.225273 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:01.278770 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:01.317737 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:01.378089 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:01.393214 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:01.396184 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:01.478229 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:01.505472 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:01.516624 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:01.552852 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:01.556510 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:01.668824 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:01.719568 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:01.757615 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:01.880884 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:01.979507 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.006902 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.013969 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:02.062489 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.068576 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:02.072944 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:02.100843 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.105485 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:02.139250 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:02.228900 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.255872 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:49:02.259989 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:02.272872 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.276613 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.287408 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:02.308379 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.324230 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.333339 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:02.360625 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:02.408772 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:02.457876 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:02.463524 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.510942 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:02.527254 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:02.541918 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:02.587807 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.610839 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:02.611733 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.654963 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:02.738184 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.750975 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:02.767870 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.783687 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:02.813100 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.818708 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.938352 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.957598 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:02.959195 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:02.979481 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:03.015857 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:03.029937 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:03.103914 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:03.112250 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:03.143919 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:03.254320 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:03.352140 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:03.368686 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:03.418624 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:03.456172 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:03.464687 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:03.538683 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:03.539217 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:03.616132 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:03.641639 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:03.665825 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:03.684927 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:03.696213 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:03.777240 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 20:49:03.818346 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:03.848826 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:03.900472 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:03.938187 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:03.957618 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:04.024870 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 20:49:04.044369 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:04.077841 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:04.087922 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:04.103312 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:04.111054 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:04.123422 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:04.163065 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:49:04.227136 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.249868 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.295585 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:04.337903 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.338793 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.410324 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:04.466734 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:04.472347 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:04.479748 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:04.556684 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.611763 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:49:04.630860 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.670034 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.682677 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.707449 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.710951 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:04.716079 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:04.760792 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:04.780864 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.826896 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:04.860687 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:04.895776 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:04.911124 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:04.917094 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.936138 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:04.978911 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:04.993810 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:05.009029 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:05.052192 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:05.107273 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:05.113592 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:05.119860 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:05.179251 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:49:05.217113 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:05.231482 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:05.255074 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:05.271845 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:05.328804 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:05.402354 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:05.408099 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:05.494243 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:05.523664 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:05.618530 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:05.643886 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:49:05.667397 25597 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:05.667599 25597 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:49:05.668596 25597 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:49:05.669222 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:49:05.669466 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:49:05.670015 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:05.670111 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:05.670147 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:49:05.670301 25597 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:05.674816 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:05.682996 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:05.690996 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:05.764199 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:05.810933 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:05.843788 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:05.896047 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:05.903790 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:05.918640 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:06.020380 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:06.056667 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:06.072798 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:06.099328 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:06.184478 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:06.187981 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:06.223758 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:06.275357 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:06.328188 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:06.403630 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:06.439106 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:06.478817 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:06.483245 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:06.486305 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:06.533747 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:06.578315 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:06.599453 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:06.606456 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:06.640168 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:06.658601 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:06.664803 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:06.713896 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:06.780913 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:06.784188 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:49:06.797694 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:06.811056 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:06.924314 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:06.962582 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.001448 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:07.006739 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.042057 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:07.090108 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:07.181288 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:49:07.188863 25880 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:49:07.189033 25880 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:49:07.189867 25880 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:49:07.190702 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:49:07.190855 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.191012 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:07.191164 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:07.191254 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:49:07.191701 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:49:07.191874 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:07.191903 25880 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:49:07.253609 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.273696 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.325865 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.356788 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.368089 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.389305 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.407349 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:07.458397 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:07.488416 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:07.562492 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:07.575048 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:07.586992 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.606627 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.606693 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:07.628356 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:07.630721 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.634990 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:07.727977 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:07.744809 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:07.767776 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.803479 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:07.852574 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:07.905723 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:07.950840 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:08.059576 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:08.107043 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:08.160334 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:08.164345 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:08.187520 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:08.205749 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:08.249423 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:08.269140 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:08.308017 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:08.308184 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:08.380028 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:08.419318 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:08.449728 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:08.487273 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:08.638037 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:08.659863 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:08.684221 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:49:08.704631 25880 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:49:08.704789 25880 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:49:08.705523 25880 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:49:08.706198 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:08.707161 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:49:08.707479 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:08.707667 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:49:08.707831 25880 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:49:08.708197 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:08.708635 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:08.765611 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:08.780153 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 20:49:08.780969 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:08.792583 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:08.829111 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:08.837503 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:08.875938 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:08.886302 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:08.952903 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:49:09.096590 26260 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:09.096765 26260 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:49:09.097483 26260 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
W20240502 20:49:09.097694 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 139 similar messages]
I20240502 20:49:09.097898 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:49:09.098080 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:09.098606 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:09.098786 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:09.098860 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:49:09.099041 26260 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:09.107442 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.126363 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.132530 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:09.150516 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:09.174248 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240502 20:49:09.182551 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:09.192119 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.247376 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:09.255256 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.281984 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.319645 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:09.343120 26262 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:09.343319 26262 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:49:09.343840 26262 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:49:09.344359 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:49:09.344555 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:49:09.345224 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:09.345330 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:09.345360 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:49:09.345508 26262 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:09.387359 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:09.409577 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:09.440021 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:09.444339 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:09.463905 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.511868 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:49:09.529258 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:09.627467 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.639055 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:09.661113 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.684697 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:49:09.699941 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:09.767189 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.787757 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:09.854082 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:09.862893 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.885641 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.935281 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:09.945500 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.987099 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:09.992671 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:10.020018 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:10.020102 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:10.071266 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:10.077850 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:10.083600 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:10.145030 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:10.152781 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:10.158094 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:10.171653 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:10.182896 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:10.204089 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:49:10.278312 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:10.296972 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:10.311525 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:10.409389 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:10.491941 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:10.496814 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:10.499137 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:10.499107 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:10.605024 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:10.691380 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:10.704422 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:10.744776 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:10.830281 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:10.913774 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:10.921908 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.025521 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:11.033340 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:11.037906 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:11.060220 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.068810 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.163925 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:11.183403 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:11.203327 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:11.218429 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.281575 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:11.288857 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:11.302384 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:11.425479 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:11.427399 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:11.469010 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.495147 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.586093 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:11.633909 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.661119 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:11.698232 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.699987 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.703832 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:11.746371 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:11.746737 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:11.754252 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.781894 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:11.785195 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:11.923698 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:12.006183 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:12.017278 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:12.050426 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.069857 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.086592 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.126317 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.170835 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.194737 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:49:12.216790 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:12.256819 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:12.280161 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:12.293479 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.304355 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.306682 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:12.361289 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:12.452834 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.468567 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:12.480712 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:12.480713 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:12.518225 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.571866 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:12.599884 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:12.616981 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.624919 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.625195 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:12.646914 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 20:49:12.686434 26359 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:49:12.686605 26359 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:49:12.687122 26359 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:49:12.688637 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:12.689680 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:12.689846 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:49:12.690301 26359 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:12.714975 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:12.715512 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:12.761657 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:12.781580 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.876396 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:12.898635 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:12.929782 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.951197 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:12.967615 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:13.045415 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:13.171929 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:13.246870 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:13.251443 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:13.297462 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:13.321780 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:13.338299 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:13.402208 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:13.431309 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:13.465826 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:13.552076 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:13.658923 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:13.692322 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:13.727958 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:13.749226 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:13.784292 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 20:49:13.814980 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:13.847088 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:13.894704 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:13.938596 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:13.966802 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:13.972762 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:13.988601 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:14.010218 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.024518 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:14.040315 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.059249 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.092305 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.120965 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 20:49:14.146498 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.148913 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:49:14.165840 27431 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:14.166270 27431 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:49:14.167151 27431 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:49:14.167924 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:49:14.168097 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:49:14.168529 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:14.168830 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:14.168900 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:49:14.169047 27431 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:14.171286 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.207803 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:49:14.259307 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:14.266645 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.297328 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:14.310240 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:14.344627 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.377535 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:14.453040 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.495889 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:14.542528 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:14.593501 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.599274 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.614462 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:14.616986 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.714965 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.728394 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:14.735978 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:14.821327 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.832453 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:49:14.835253 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:14.850257 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:14.865442 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.869973 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.899580 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:14.919283 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.944072 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:14.981668 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:14.983740 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:14.986665 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:15.001042 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:15.093612 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:15.110620 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:15.131315 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:15.151871 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:15.204092 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:15.232192 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:49:15.294062 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:15.360762 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:15.376250 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:15.377481 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:15.387461 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:15.393266 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:15.400148 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:15.508827 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:15.549911 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:15.584720 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:15.611964 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:15.618288 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:15.675412 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:15.715667 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:15.833312 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:15.861313 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:15.913878 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:15.947141 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:15.986702 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:16.022568 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.169661 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:16.191730 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:16.273891 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:16.293094 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:16.298436 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:16.369307 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.406677 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.412400 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:16.480019 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:16.486788 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.513391 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:16.520545 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:16.540576 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.571807 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:16.612015 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.624089 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:16.628628 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:16.674559 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.707175 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.754861 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:16.786201 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.817741 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.859036 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:16.890612 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:16.892606 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:16.981626 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:17.000267 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:17.047626 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:17.053546 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.094494 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.121827 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.275972 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:17.282599 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:17.355208 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:17.387321 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:17.404304 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.405000 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.406746 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:49:17.425860 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.428112 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.484246 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:17.497171 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.500954 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:17.552001 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.579900 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:17.582474 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.608045 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:49:17.633965 28160 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:17.634176 28160 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:49:17.634932 28160 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:49:17.635979 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:49:17.636283 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:17.636575 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:17.637279 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:17.637403 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:49:17.637717 28160 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:17.649132 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.658739 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:17.671011 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:17.712530 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:17.806607 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.830000 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:17.884308 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:17.918433 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:18.033299 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:18.046006 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:18.053987 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:18.085134 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:18.109885 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:18.147145 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:18.173439 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:18.206591 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:18.210045 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:18.231552 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:18.256515 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:18.298938 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:18.408154 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:18.575661 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:18.616310 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:18.624826 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:18.635212 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:18.647670 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:18.698750 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:18.750659 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:18.793676 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240502 20:49:18.794699 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:18.835949 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:18.896522 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:18.953771 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:18.977946 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.003244 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:19.011585 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:19.013273 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:19.118927 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:19.127156 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 20:49:19.128008 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.128057 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:19.130920 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:19.141297 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.165428 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:19.169601 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.232918 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.252698 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 20:49:19.270607 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:19.276065 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:19.445421 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:19.489691 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:19.509408 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.538228 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:19.574666 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.589601 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.609237 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.651553 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.653122 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.668711 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:19.722685 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:19.729382 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.779978 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:19.871186 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.908453 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.913769 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:19.936504 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:19.953845 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:20.051096 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.059307 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:49:20.091405 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.094592 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:20.109423 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:20.138550 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.154572 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:20.174248 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.191365 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:20.192111 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.195950 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.246368 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:20.250625 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:49:20.260882 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:20.274471 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:20.292569 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.423759 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.440872 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.501958 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:20.574887 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.585577 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:20.599440 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:20.634846 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:20.662276 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.726895 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:20.734225 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:20.764048 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:20.788347 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:20.877624 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:20.961293 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:21.156800 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:21.161834 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:21.176612 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:21.180606 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:21.270676 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:21.301349 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:21.351677 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:21.358791 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:21.466879 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:21.475879 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:21.512764 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:21.550805 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:21.573123 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:21.595937 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:21.605441 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:21.661948 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:21.668144 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:21.673019 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:21.698557 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:21.733803 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:21.757680 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:21.769038 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:21.816634 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:21.900509 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:21.908883 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:21.946029 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:21.957222 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:21.965992 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:21.967599 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.100402 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.126289 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:22.148424 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:22.194113 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.229483 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.243444 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.272517 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:22.295827 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.295995 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.312984 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.383204 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.389365 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:22.459863 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.470911 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.511699 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:22.518419 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:22.525440 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:22.528470 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:22.550282 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:22.697176 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:22.699815 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:22.709475 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:22.711544 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:49:22.762066 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:22.817834 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:22.822058 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:22.843811 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:22.867143 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.925372 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:22.965672 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:22.995561 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:23.046741 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:23.050379 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:23.051352 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:23.053087 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:23.128832 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:23.151319 28537 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:49:23.151525 28537 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:49:23.152355 28537 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:49:23.152844 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:49:23.153649 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:23.153788 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:49:23.154268 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:49:23.154433 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:23.154434 28537 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:49:23.156008 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:23.167984 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:23.183985 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:23.203187 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:23.436036 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:23.507763 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:23.546319 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:23.551071 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:23.596689 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:23.598374 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:23.598420 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:23.700594 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:23.773175 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:23.848351 29067 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:23.848570 29067 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:49:23.849249 29067 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:49:23.849797 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
W20240502 20:49:23.849972 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 183 similar messages]
I20240502 20:49:23.850001 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:49:23.850687 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:23.850808 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:23.850852 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:49:23.850989 29067 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:23.981637 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:23.986546 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:23.990659 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:23.998214 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:23.999899 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.034765 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.039973 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:24.045644 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.055688 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:24.094431 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:24.096072 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:24.130138 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240502 20:49:24.130893 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.159842 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:24.243541 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.249219 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:24.280851 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:49:24.281920 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:24.425942 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.436631 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:24.445161 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.445396 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.493338 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:24.500237 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.512979 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:24.545266 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.619741 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:24.620676 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:24.672545 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.681973 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.720175 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:24.721015 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.724673 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.758298 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.771550 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.784288 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.885170 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:24.916364 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:24.971704 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:24.979187 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:24.994096 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:49:25.005627 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:25.049976 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:25.051837 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:25.088979 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:25.100049 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:25.110715 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:25.146342 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:25.160095 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:25.200312 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:25.271387 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:49:25.272459 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:25.299196 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:25.310271 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:25.316423 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:25.328248 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:25.336699 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:49:25.362385 29560 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:49:25.362608 29560 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:49:25.363327 29560 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:49:25.363755 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:25.365316 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:49:25.365880 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:25.366014 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:49:25.366283 29560 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:49:25.366441 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:25.452513 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:25.496623 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:25.552652 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:25.575045 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:25.577895 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:25.578869 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:25.748263 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:25.789906 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:25.795150 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:25.976898 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:26.016167 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.025813 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.050025 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:26.070881 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:26.128794 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:26.138008 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.220963 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.235074 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:26.267006 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:26.479491 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.502861 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:26.515060 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:26.522220 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.558650 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:26.624531 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:26.631458 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:26.631630 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:26.641001 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:26.672500 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:26.676010 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:26.693413 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:26.759335 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.842270 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.848749 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:26.860055 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.944077 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.955466 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.960685 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:26.985163 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.063089 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.076879 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:27.112875 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:27.129407 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:27.151342 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:27.167874 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.199200 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.203548 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:27.211097 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.216651 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.239813 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:27.250855 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.366147 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:27.406886 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.418857 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:27.419832 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.525959 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:27.541546 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.580255 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.583766 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:27.606526 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:27.628388 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:27.660738 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:27.682844 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:27.741360 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:49:27.741799 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:27.750953 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:27.756982 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:27.765909 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.801541 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:27.896864 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:27.924189 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:27.962960 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:27.976593 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:28.058214 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:28.174711 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:28.238560 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:28.279511 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:28.295341 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:28.313462 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:28.348423 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:28.418427 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:28.444022 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:28.444383 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:28.607357 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:28.622748 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:28.721827 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:28.751111 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:28.792037 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:28.797959 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:28.835362 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:28.844344 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:28.885571 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 187 similar messages]
W20240502 20:49:28.916450 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:28.964149 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:49:29.006274 28537 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:49:29.006500 28537 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:49:29.007158 28537 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:49:29.008810 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:29.009048 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:49:29.009203 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:49:29.009389 28537 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:29.033269 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:29.040731 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:29.047515 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.050359 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:29.057248 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:29.071112 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:29.200793 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:29.207890 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 20:49:29.215895 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:29.238430 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:29.243424 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.307567 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.333763 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 20:49:29.351872 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:29.387519 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.404748 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.440310 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:29.493674 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:29.497967 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:29.554387 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.594669 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:29.606298 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.625236 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.626233 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:29.635354 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.646409 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.650357 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:29.659302 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.684019 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.712965 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.809864 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:29.833767 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:29.853317 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:29.878966 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:29.932602 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:29.954260 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:29.954669 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:30.042554 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.045411 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:30.053953 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:30.067286 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:30.074913 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.104996 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:49:30.166471 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:30.183593 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.216549 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:30.350267 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:30.350354 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:30.365620 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.404155 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.420791 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:49:30.440310 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:30.471715 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.482427 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:30.510347 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.682075 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:30.693318 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:30.718905 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.740829 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.757241 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:30.793198 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.869565 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:30.872411 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.905189 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:30.924335 30403 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:49:30.924530 30403 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:49:30.925269 30403 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:49:30.925720 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:30.926177 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:49:30.926492 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:30.926676 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:49:30.926867 30403 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:49:30.926882 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:30.938769 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:30.955554 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:30.987360 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:31.047539 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:31.053681 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:31.300187 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:31.315358 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:31.354001 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:31.408929 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:31.466827 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:31.498752 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:31.562005 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:31.562424 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:31.598187 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:31.602527 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:31.711864 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:31.777787 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:31.810824 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:31.816231 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:31.831107 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:31.839710 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:31.855048 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:31.871515 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:31.887006 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:31.891914 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:31.908785 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:31.932289 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:31.953320 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:32.010238 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.103590 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.123966 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:32.174079 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.185652 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.205307 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:32.208580 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.209476 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.214219 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.223207 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.260672 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:32.298161 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:32.330318 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:32.371210 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.393142 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:32.432303 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:32.493407 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.519973 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:32.535445 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:32.575177 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.615026 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:49:32.638986 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:32.693114 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:32.698091 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:32.844105 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:32.849881 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:32.874545 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:32.880190 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:32.931685 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:49:32.951941 30403 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:32.952211 30403 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:49:32.952962 30403 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:49:32.953579 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:49:32.953852 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:32.954609 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:32.954764 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:32.954854 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:49:32.955040 30403 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:32.956532 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:33.021376 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:33.031646 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:33.055907 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:33.090762 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:33.100400 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:33.135010 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:33.156564 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:33.165772 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:33.217027 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:33.226140 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:33.291745 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:33.337893 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:33.338009 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:33.367272 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:33.470391 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:33.486945 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:33.661768 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:33.689457 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:33.830101 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:33.839148 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:33.902060 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:33.924991 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:33.939637 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 178 similar messages]
W20240502 20:49:33.940965 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:33.957327 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:33.980901 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.029505 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:34.134963 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.157543 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.175143 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.191519 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:34.211855 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 20:49:34.230693 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:34.276204 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:34.327481 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:34.360442 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 20:49:34.361275 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.379209 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:34.392560 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.411996 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:34.453909 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:34.465553 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.469695 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:34.493698 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:34.529301 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.538960 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.591084 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.605618 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.625778 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.636667 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.676535 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:34.821877 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.835086 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:34.839959 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:34.848754 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.853665 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:34.858280 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:34.869853 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:34.888295 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:34.920334 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:34.945220 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:34.948117 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:49:34.975001 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:35.105137 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.121244 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:35.156265 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.161576 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:35.330785 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.376863 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:35.416327 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:35.431653 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:35.434072 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:35.435751 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 20:49:35.464079 30967 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:35.464264 30967 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:49:35.464960 30967 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:49:35.465715 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:35.465927 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:35.466835 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:35.466979 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:35.467042 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:49:35.467240 30967 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:35.470160 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:35.530550 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.551270 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:49:35.566392 30967 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:35.566604 30967 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:49:35.567183 30967 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:49:35.567886 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:49:35.568125 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:35.568617 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:35.568861 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:35.568926 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:49:35.569128 30967 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:35.582955 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.643121 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.665983 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.702724 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.707823 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.798662 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.812568 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.818703 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:35.820164 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:35.884739 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:35.887495 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:35.910784 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:35.913100 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:35.954869 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:36.076761 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.089251 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:36.198534 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.322350 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:36.375371 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:36.412619 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:36.434629 30967 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:49:36.434803 30967 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:49:36.435459 30967 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:49:36.435998 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:36.436312 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:49:36.436746 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:36.436820 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:49:36.436960 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:36.436980 30967 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:49:36.475371 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.481335 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:36.522981 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:36.530102 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:36.564635 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:36.619380 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.632651 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:36.660058 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.694418 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:36.712850 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.729908 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:36.730527 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.765874 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.801714 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:36.857023 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.866639 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.874758 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:36.899768 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:36.904145 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:36.954793 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:36.985313 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:37.026643 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:37.067615 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:37.080920 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:37.103044 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:37.109609 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:37.162685 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:37.184281 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:37.194986 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:37.209784 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:37.256719 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:37.263723 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:37.333112 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:37.342303 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:37.397732 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:49:37.448777 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:37.458810 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:37.516085 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:37.517499 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:37.519215 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:37.540024 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:37.560653 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:37.610339 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:37.689208 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:37.741204 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:37.870874 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:37.888947 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:37.892421 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:37.905123 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:37.935859 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.062258 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.070235 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:49:38.076139 31777 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:38.076407 31777 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:49:38.077311 31777 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:49:38.077838 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:49:38.078039 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:49:38.078795 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:38.078963 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:38.079080 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:49:38.079274 31777 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:38.157719 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.200457 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.202762 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:38.206530 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:38.227861 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.275110 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:38.277215 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.279652 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:38.312857 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.316134 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.344990 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:38.347311 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:38.439532 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:38.618315 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.639216 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:38.678637 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:38.741945 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:38.795922 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:38.838958 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.842247 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:38.859275 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:38.882980 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:38.889593 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.001669 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 20:49:39.004666 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:39.040040 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:39.132697 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.172322 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.181576 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.217200 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:39.238006 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:39.245437 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:39.249894 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.264912 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.266685 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 20:49:39.295820 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:39.298065 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:39.304654 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:39.361966 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240502 20:49:39.412806 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:39.425009 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.458891 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:39.470969 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:39.477128 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.491935 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.530050 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.536083 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.570672 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.577538 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:39.681782 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:39.690925 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.720656 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:39.728639 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:39.753969 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:39.793972 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:39.801043 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:49:39.849860 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:39.920972 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:39.943149 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:39.970028 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:40.062778 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:40.094970 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:40.098637 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.136536 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:40.230329 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:40.276911 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.280339 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:40.282166 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.342940 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:40.371729 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:40.417148 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.438597 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.450238 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:40.506983 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.561319 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.576254 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:49:40.611825 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.615914 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.637511 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.658627 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:40.685657 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.761132 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:40.807121 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:40.845933 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:40.883342 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:40.888330 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:40.891088 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:40.914363 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:49:41.111850 32212 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:49:41.112080 32212 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:49:41.113008 32212 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:49:41.113765 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:41.113888 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:49:41.114903 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:41.115056 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:49:41.115270 32212 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:49:41.115396 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:41.138756 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:41.172335 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:41.173230 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:41.207422 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:41.257432 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:41.283175 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:41.292814 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:41.346832 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:41.373414 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:41.382411 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:41.492321 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:41.543430 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:41.545998 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:41.589308 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:41.648705 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:41.738427 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:41.755052 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:41.772817 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:41.798445 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:41.825207 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:41.832950 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:41.845580 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:41.859721 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:41.888160 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:41.927790 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:41.933982 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:41.952323 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:41.974661 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.039184 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.061332 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.092496 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.101760 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.113773 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.124503 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:42.162686 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.164881 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.222672 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:42.273505 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:42.279739 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:42.351040 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:42.352823 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.385453 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:49:42.399516 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:42.498245 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:42.511399 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:42.574481 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.623304 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.714087 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:42.732985 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:42.763527 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:42.814352 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:42.859618 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:42.907523 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:42.923763 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:42.941152 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:42.946974 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:42.970337 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:43.061269 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.082186 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:43.083561 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.087812 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.089605 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.113104 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:43.113529 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.127147 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.195184 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:43.310286 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:43.395159 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:43.445843 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:43.496062 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.507079 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:43.507256 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.518877 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.648481 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:43.649821 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.653810 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:43.842859 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:43.849805 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.865808 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:43.878077 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:43.879469 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:43.906870 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:43.931627 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:44.010910 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 20:49:44.068940 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:44.091383 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.099491 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:44.109722 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:44.198539 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.278782 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:44.308140 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240502 20:49:44.309494 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:44.348716 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.351411 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:44.364293 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.370095 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:44.372565 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 20:49:44.373648 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:44.380159 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:44.404865 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:44.404868 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.436805 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.524065 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.594108 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.614805 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:44.648830 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.666829 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:49:44.687582   389 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:44.687754   389 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:49:44.688269   389 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
W20240502 20:49:44.688788 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:49:44.689898 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:44.690068 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:44.690621 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:44.690735 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:44.690778 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:49:44.690920   389 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:44.733441 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.748515 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.764851 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:44.765141 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.774264 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.848809 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:44.902012 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:49:44.962318 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:44.962443 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:44.982743 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:44.999334 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:45.104943 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.183913 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:45.188163 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:45.311784 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:45.342644 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:45.374675 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:45.384524 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.401517 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.408023 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.448297 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.485591 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.542702 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.556841 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:45.558568 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:45.559274 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.560005 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:45.560698 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.571779 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:45.622298 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.632099 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:49:45.779294 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.845800 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:45.846278 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:45.848006 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:45.859185 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.909924 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:45.914976 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:49:45.936653   389 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:49:45.936821   389 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:49:45.937319   389 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:49:45.938593 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:45.939780 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:45.939885 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:45.939944 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:49:45.940276   389 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:45.952594 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:45.976367 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:46.005975 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:46.290925 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:46.328714 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:46.352548 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:46.371686 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:46.393028 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:46.447798 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:46.501539 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:46.588220 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:46.609280 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:46.612450 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:46.613468 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:49:46.712262   389 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:46.712435   389 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:49:46.713001   389 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:49:46.713506 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:49:46.713708 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:46.714205 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:46.714555 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:46.714630 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:49:46.714848   389 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:46.756414 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:46.787736 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:46.806489 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:46.811198 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:46.875782 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:46.888413 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:46.922299 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:46.942780 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:46.944981 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:46.975841 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:46.993499 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:47.009387 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:47.013492 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:47.035312 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:47.041186 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:47.050052 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:49:47.054199 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:47.085358 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:47.206604 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:47.210464 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:47.223448 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:47.241932 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:47.351996 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:47.399713 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:47.405884 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:47.423373 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:47.430855 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:47.443643 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:47.463514 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:47.464493 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:47.493736 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:47.570282 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:47.599362 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:47.618041 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:47.714015 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:47.835572 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:47.837615 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:47.866305 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:47.897222 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:47.912606 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:47.975150 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:47.988668 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:47.991217 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:47.998245 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:48.076892 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:48.120390 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:48.133149 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:48.164805 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:48.180531 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:48.220412 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:48.286453 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:48.386157 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:48.401757 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:48.447150 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:48.454218 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:48.459333 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:48.536108 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:48.542032 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:48.602368 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:48.734802 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:48.761590 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:48.829507 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:48.882941 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:48.913550 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:48.942554 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:48.973479 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:48.990749 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:49.030009 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:49.105538 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 184 similar messages]
W20240502 20:49:49.110392 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:49.129772 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.167879 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.227000 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.282938 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:49.293162 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.300012 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.343801 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 20:49:49.355921 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:49.358634 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.364253 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.387610 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:49.400518 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 20:49:49.401520 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:49.401600 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:49.449401 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:49.469265 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:49:49.515313 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.528769 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.541226 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:49.637645 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:49.647105 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:49.671618 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.677951 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.699270 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.753968 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.778404 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:49.875623 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:49.916991 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.928874 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:49.932363 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:49.940853 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:49.949622 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:50.011610 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.067889 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.068396 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.075522 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:49:50.093806  1310 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:50.093991  1310 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:49:50.094538  1310 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:49:50.094903 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:50.095105 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:50.096058 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:50.096207 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:50.096282 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:49:50.096446  1310 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:50.136370 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:50.167086 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.190291 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:49:50.245560   719 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:50.245816   719 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:49:50.246608   719 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:49:50.247190 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:49:50.247542 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:49:50.248080 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:50.248206 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:50.248309 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:49:50.248481   719 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:50.252373 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.295827 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:50.301357 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.339131 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.420648 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.427098 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:50.435220 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:50.438158 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:50.454195 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.512368 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.527993 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:50.608059 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.645974 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:49:50.719230 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:50.738538 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:50.814477 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:50.846453 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:50.861459 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:50.887898 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:50.934888 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:50.986974 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:51.033713 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.043010 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.067551 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.221562 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:51.335816 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.344033 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:51.366549 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.367797 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:51.374675 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:51.391553 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:51.462594 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.483276 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:51.492533 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:51.620676 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:51.634994 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.641729 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.644834 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.692447 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:51.699252 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.727080 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:51.734261 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.766355 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:51.784936 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:51.805181 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.810197 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:51.825886 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:51.874441 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:51.879711 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:51.905771 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:51.905936 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:49:51.946724 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:52.092767 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 20:49:52.142561  1487 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:52.142743  1487 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:49:52.143584  1487 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:49:52.144531 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:52.144747 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:52.145781 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:52.145896 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:49:52.145951 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:49:52.146095  1487 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:52.183729 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:52.184911 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:52.214489 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:52.276234 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:52.309839 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:52.321086 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:52.339912 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:52.350162 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:52.375128 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:52.383988 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:52.442157 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:52.513783 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:52.572163 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:52.652314 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:52.732079 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:52.748466 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:52.748723 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:52.763353 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:52.767128 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:52.770320 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:52.811928 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:52.828712 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:52.853854 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:52.934703 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:52.937602 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:52.944087 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:53.002656 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.010344 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.037176 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.064195 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.077417 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.087725 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:53.195611 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:53.228333 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:53.294728 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.335057 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:53.374795 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:53.454380 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:53.519691 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:53.522485  1487 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:49:53.522648  1487 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:49:53.523340  1487 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:49:53.523994 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:49:53.524219 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:49:53.524785 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:49:53.525138 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:49:53.525372 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:53.525388  1970 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:49:53.656322 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.657274 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:53.744159 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.744325 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:53.785692 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.806960 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:53.910203 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.911381 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:53.936475 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:53.949469 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.956836 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:53.986616 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:54.035116 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:54.038749 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:54.075755 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:54.082814 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:54.115027 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:54.138559 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:54.141841 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:54.156668 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:49:54.219683 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:54.245391 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:54.261143 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:54.298486 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:54.299391 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:54.313016 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:54.317981 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:54.362191 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:54.424429 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 20:49:54.425540 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:49:54.478065 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 20:49:54.496840 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:54.518772 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:54.581781 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:54.581907 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:54.613986 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:54.692787 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:54.756826 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:54.830691 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:54.842288 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:54.862686 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:54.974494 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.033353 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:55.039400 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:55.044350 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.061609 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:55.111012 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:55.146270 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:55.157953 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.163179 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:55.174893 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:55.176049 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.194690 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:55.211609 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:55.222649 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.247448 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:55.343981 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:55.354688 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.378193 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.452220 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:55.488039 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:55.522320 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:55.643193 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.675233 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:49:55.683073 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:55.689958 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.699213 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.725503 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:55.755985 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:55.775766 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.778843 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.799923 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:55.819954 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:55.921342 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:56.041811 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.091401 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:56.129930 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:56.183957 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:56.202375 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:56.238159 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.241853 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.289764 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.376049 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:56.403558 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.409318 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.415179 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:56.498997 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:56.521698 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.530951 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.564862 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.586230 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.589895 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:56.589996 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.607898 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:56.621927 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:56.639927 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:56.661831 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:56.695362 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:56.726706 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:56.785478 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:56.797780 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:56.812685 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:56.879195 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:56.998287 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:49:57.027318 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:57.061198 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:57.077900 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:57.120354 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:57.151628 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:57.182628 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:57.238992 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:57.309962 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:57.351578 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:57.442731 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:57.443939 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:57.457422 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:57.464426 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 20:49:57.540656  2533 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:49:57.540867  2533 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:49:57.541493  2533 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:49:57.542759 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:57.542870 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:49:57.542941 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:49:57.543138  2533 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:57.585301 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:49:57.587118  2533 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:49:57.587267  2533 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:49:57.587883  2533 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:49:57.588618 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:49:57.588819 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:49:57.589447 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:57.589708 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:49:57.589790 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:49:57.589968  2533 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:49:57.594624 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:57.603688 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:57.622078 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:57.627518 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:57.692135 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:57.701560 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:57.735697 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:57.768333 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:57.772012 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:57.804276 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:57.830855 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:57.879225 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:57.879231 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:57.904266 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:57.908572 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:57.971119 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:57.974275 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.089965 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.104543 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:58.117048 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.169449 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.176234 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.235347 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:58.263365 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:58.294838 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:58.365579 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:58.430177 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.436252 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:58.567193 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:58.693789 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:58.721771 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.731016 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:58.745857 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.776110 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:58.810545 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:58.820318 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.823530 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.828256 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.843847 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:58.974268 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:59.006853 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:49:59.036451 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.038717 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.058840 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.098974 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:59.151979 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:49:59.152884 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:59.176668 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:49:59.177615 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:59.177699 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.180800 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:59.185875 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.204047 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.252370 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:59.259408 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:59.342682 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:59.343245 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:59.353372 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:49:59.431171 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 20:49:59.484903 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.503152 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 20:49:59.504083 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:49:59.581753 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:49:59.619378 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:49:59.657275 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.722294 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:49:59.757452 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:49:59.759364 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:59.771785 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.830081 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.883953 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:49:59.912313 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:49:59.942309 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:49:59.970135 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:00.015307 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:00.092886 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.107101 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:00.113221 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.185957 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:00.244222 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.248723 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:00.285409 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.320744 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:00.352751 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:00.363323 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:00.394595 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:00.396063 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:00.438133 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.501402 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.505941 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.508358 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:00.564462 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.649814 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.671383 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:00.678118 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.718796 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:00.727794 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:00.751065 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:50:00.776073 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:00.795231 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:00.891650 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:00.903894 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:00.960414 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.057019 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:01.057369 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.086093 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.097941 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:01.105959 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.189463 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:01.253994 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.276485 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.279121 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:01.307052 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:01.390769 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.442478 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:01.498797 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.547585 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:01.594836 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:01.622985 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:01.627902 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:01.633061 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:01.650346 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:01.693595 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:01.765902 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:01.765863 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.780016 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.780192 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:01.795565 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.851936 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:01.853475 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:01.875406 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.884806 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:01.951571 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:02.019685 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:50:02.045122 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:02.065774 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:02.146167 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:02.152737 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:02.176553 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:02.204277 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:02.222847 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:02.223268 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:02.301420 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:02.303989 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:02.381211 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:02.386304 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:02.425587 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:02.476981 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:02.477116 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:02.538733 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:02.567035 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:02.681103 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:02.692226 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:02.732045 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:02.805919 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:02.809906 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:02.837980 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:02.849836 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:02.858448 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:02.910100 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:02.922588 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:02.923529 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:02.969272 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:02.969323 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:03.018689 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:03.059265 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:03.061825 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:03.080169 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:03.095690  3604 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:03.095872  3604 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:50:03.096486  3604 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:50:03.096985 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:50:03.097237 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:50:03.097955 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:03.098331 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:03.098414 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:50:03.098603  3604 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:03.253664 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:03.334942 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:03.346521 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:03.358695 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:03.368803 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:03.440627 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:03.443948 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:03.473547 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:03.507447 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:03.553387 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:03.594740 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:03.683879 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:03.701481 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:03.757028 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:03.764379 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:03.768163 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:03.842981 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:03.868031 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:03.992914 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.003916 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.027905 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:04.047350 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:04.064749 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.081670 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:04.089648 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.149014 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:04.182516 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:04.191637 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 20:50:04.199952 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.204138 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.234743 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:04.271229 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:04.304747 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:04.307638 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:04.372275 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:04.375602 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:04.391687 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:04.469542 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240502 20:50:04.512379 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.540380 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 20:50:04.541749 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:04.646014 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.652205 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:50:04.659302 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:04.666783 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:04.678226 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:04.695937 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.700623 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.704622 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:04.724195 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.780814 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:04.821969 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:04.853819 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:04.858741 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:04.890733 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:04.963676  3604 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:50:04.963878  3604 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:50:04.964706  3604 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:50:04.965196 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:04.965776 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:50:04.967095 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:04.967168 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:50:04.967298 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:04.967363  3604 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:50:05.099800 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:05.103775 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:05.123716 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:05.143795 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:05.147047 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:05.173638 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:05.185628 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:05.218997 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:05.219889 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:05.254021 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:05.260165 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:05.331949 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:50:05.355430  3604 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:50:05.355605  3604 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:50:05.356115  3604 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:50:05.357496 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:05.357604 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:50:05.357676 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:50:05.357827  3604 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:05.385705 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:05.449613 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:05.463960 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:05.469146 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:05.589430 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:05.623288 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:05.652839 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:05.709849 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:05.744087 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:05.777499 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:50:05.818416 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:05.868412 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:05.936144 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:05.979434 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.030428 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.030480 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:06.118898 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:06.177335 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.206903 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.208781 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:06.250859 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.254408 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.272740 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:50:06.295380  3604 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:06.295560  3604 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:50:06.296114  3604 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:50:06.296630 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:50:06.296803 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:50:06.297622 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:06.297713 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:06.297783 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:50:06.297981  3604 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:06.347064 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:06.360000 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.389168 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:06.483321 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.486217 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.538005 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:06.549536 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:06.652912 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.658035 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:06.682974 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:06.765444 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:06.778136 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.822850 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:06.834154 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:06.883039 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:06.883157 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:06.884677 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:06.962091 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.977035 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:06.985637 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:07.018589 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:07.037606 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:07.048511 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:07.053124 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:07.084084 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:07.119264 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:07.127313 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:07.182106 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:07.191294 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:07.207547 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:50:07.269868 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:07.296823 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:07.320175 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:07.354200 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:07.415747 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:07.416612 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:07.430655 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:07.457654 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:07.602798 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:07.677412 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:07.746484 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:07.773293 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:07.802858 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:07.803659 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:07.853614 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:50:07.858743  4231 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:07.859016  4231 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:50:07.859742  4231 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:50:07.860371 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:07.860595 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:50:07.861646 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:07.861799 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:07.861860 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:50:07.862052  4231 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:07.905849 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:07.951409 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:07.968851 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:08.003651 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:08.023003 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:08.026342 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.082994 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:08.086110 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.118202 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.125774 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.178948 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.220692 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:08.255339 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.287283 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:08.339365 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:08.357434 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:08.365936 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.555544 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:08.562021 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.567186 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:08.591401 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:08.624933 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.655858 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:08.682438 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.696837 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.710613 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:08.719098 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.769847 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:08.886483 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:08.980427 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:09.029422 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:09.108732 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:09.147111 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:09.190438 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:09.194176 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:50:09.225596 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:09.232631 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:09.242080 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:09.285840 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:09.322465 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:09.367596 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:09.379369 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:09.391639 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:09.467110 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:09.491693 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:09.507860 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:50:09.520720  4632 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:09.520920  4632 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:50:09.521550  4632 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:50:09.522389 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:50:09.522619 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:50:09.523090 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:09.523180 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:09.523283 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:50:09.523480  4632 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:09.536181 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 20:50:09.581187 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:09.594050 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:50:09.623622 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:09.624533 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:50:09.625265 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:09.647157 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:09.676952 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:09.681360 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:09.688352 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:09.693393 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:09.712884 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:50:09.730759 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:09.772626 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:09.788372 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:09.817137 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:09.829921 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:09.835101 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:09.862804 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:09.877220 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.004508 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:10.069504 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:10.069983 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:10.094100 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:10.109242 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.204103 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:10.219561 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:10.222157 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.301302 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:10.333154 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.376397 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.457388 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:10.457787 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:10.509572 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:10.509593 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:10.518826 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:10.556423 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.577484 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.611590 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.641496 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.655647 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.712234 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:10.715845 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:10.771907 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.847741 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:10.890554 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:10.932953 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:11.037794 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.043972 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:50:11.054960 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:11.055197 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:11.060061 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:11.118650 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:11.175264 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.262856 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.268680 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.287976 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.303326 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:11.435112 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.514364 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:11.532301 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:11.543285 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:11.580826 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.582911 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:11.603199 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:11.679461 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.752177 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.788499 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:11.885331 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:11.911535 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.918313 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:11.938669 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:11.943076 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:11.956050 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:11.980167 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:11.980253 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:11.992146 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:12.000423 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:12.015550 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:12.035496 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:12.068429 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:12.169945 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:12.255954 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:12.272082 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:12.283620 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:12.303963 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:50:12.315225 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:12.320288 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:12.320281 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:12.330576 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:12.362709 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:12.364961 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:12.367957 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:12.412655 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:12.453444 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:12.523440 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:12.540103 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:12.554356 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:12.558890 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:12.678818 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:12.710544 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:12.735487 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:12.752177 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:12.783932 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:12.948153 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:12.961185 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:12.969202 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.001806 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.026484 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:13.047276 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:13.060415 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:13.146502 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.161211 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:13.219696 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.249292 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:13.271749 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.349799 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:13.352839 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:13.353349 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:13.430164 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.442760 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.480520 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.509443 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:13.544984 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:13.560725 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.657469 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.729147 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.730494 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:13.752969 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.753635 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:13.927690 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:13.945195 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:13.968726 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:14.031004 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.115088 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:14.131222 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:14.165122  5050 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:50:14.165387  5050 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:50:14.166311  5050 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:50:14.166693 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:14.167618 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:50:14.168607 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:14.168697 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:50:14.169004 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:14.169550  5430 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:50:14.217530 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:14.232479 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:50:14.245401 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.254897 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:14.263969 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.277395 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:14.321215 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 20:50:14.342782  5429 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:14.342978  5429 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:50:14.343601  5429 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:50:14.344322 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:50:14.344631 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:50:14.345541 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:14.345669 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:14.345733 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:50:14.345975  5429 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:14.346449 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.422875 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:14.437474 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:14.437971 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.445802 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.470836 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.505602 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:14.513171 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:14.534322 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:14.555442 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:14.561108 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240502 20:50:14.562073 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:14.625610 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 20:50:14.626575 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.632262 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.647413 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:14.667966 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.669508 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:14.678967 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:14.752272 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:14.810407 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:14.840430 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:14.905701 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:14.915315 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:14.960609 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:15.016633 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:50:15.026684 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:15.074721 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:15.114377 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.132817 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.143522 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:15.183838 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.307574 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:15.316532 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.317467 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:15.332643 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:15.346295 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:15.426069 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.439378 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:15.457671 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.459955 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:15.519907 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:15.594533 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:15.638077 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.841984 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.873363 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.894896 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:15.923118 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:15.927387 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:15.929553 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:15.930011 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:15.946399 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.954115 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:15.992162 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:16.081584 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:50:16.127993 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:16.246196 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:16.297776 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:16.338773 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:16.372582 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:16.379601 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:16.413544 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:16.419958 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:16.465287 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:16.470233 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:16.515726 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:16.543875 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:16.565611 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 20:50:16.629817  5714 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:50:16.630043  5714 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:50:16.630821  5714 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:50:16.631470 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:16.631650 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:50:16.632447 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:16.632603 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:50:16.632700 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:16.632824  5714 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:50:16.633998 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:16.640146 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:16.683022 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:16.714466 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:16.801136 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:16.816887 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:16.861719 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:16.869889 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:16.891512 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:16.958238 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:16.985018 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:17.001801 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:17.022850 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:17.024149 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:17.042133 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:17.052621 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:17.060777 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:17.063521 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:17.078367 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:17.102406 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:17.186830 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:17.241693 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:50:17.265494 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:17.269804 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:17.295593 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:17.372128 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:17.374469 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:17.443735 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:17.473363 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:17.488885 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:17.491834 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:17.560331 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:17.585412 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 20:50:17.606225  5714 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:50:17.606434  5714 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:50:17.607012  5714 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:50:17.608590 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:17.608722 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:17.608829 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:50:17.609117  5714 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:17.611207 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:17.625238 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:17.629667 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:17.725489 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:17.791101 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:17.874095 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:17.918076 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:18.015794 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:18.019980 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:18.052587 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:18.076196 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:18.079780 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:18.113917 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:18.121742 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:18.137948 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:18.179107 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:18.220630 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:18.222749 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:18.311416 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:18.422302 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:18.437875 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:18.491325 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:18.560106 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:18.568621 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:18.670475 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:18.801687 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:18.856799 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:18.856884 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:18.892488 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:18.894671 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:18.912143 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:18.970243 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:18.995150 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.038794 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.054803 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.055509 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:19.085976 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:19.153664 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.156672 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:19.173858 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.196774 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.226578 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.233457 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:50:19.271749 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:19.341480 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:19.352001 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:19.359313 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:19.381219 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.386349 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:19.429853 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:19.462903 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:19.497767 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.503583 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:19.503990 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.569700 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 20:50:19.570734 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:19.585281 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:19.586761 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:19.673874 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240502 20:50:19.690775 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:19.706946 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.724525 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.789947 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:50:19.795164 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:19.823714 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:19.849732 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:19.872505 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:19.874923 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:19.898478 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.942070 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:19.951208 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:20.043742 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:20.053730 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:20.056882 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:20.155172 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:20.263780 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:20.288486 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:20.303334 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:20.341312 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:20.343183 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:20.359247 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:20.376302 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:20.440330 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:20.508236 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:20.511381 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:20.563364 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:50:20.604863  6379 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:20.605032  6379 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:50:20.605787  6379 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:50:20.606532 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:50:20.606695 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:50:20.606863 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:20.607825 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:20.607939 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:50:20.608180  6379 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:20.675647 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:20.730485 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:20.742426 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:20.750275 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:20.752665 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:20.791477 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:20.792238 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:20.864063 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:20.884742 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:20.939410 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:21.050573 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:21.169859 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:50:21.199700 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:50:21.213829  6544 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:21.214018  6544 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:50:21.214766  6544 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:50:21.215930 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:21.216143 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:50:21.217535 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:21.217677 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:21.217725 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:50:21.217911  6305 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:21.240345 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.296804 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:21.306877 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.311415 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:21.313611 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.335373 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.390753 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:21.397042 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.410786 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:21.442660 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:21.458731 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:21.502910 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.519590 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.534636 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.582350 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:21.602734 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:21.618211 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:21.636708 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:21.648731 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.737649 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:21.743211 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.753592 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:21.779827 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.812934 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:21.834914 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.835047 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:21.840956 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:21.857565 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:21.930228 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:21.973111 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:22.006901 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:22.083912 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:22.101228 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:22.174831 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:22.207077 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:50:22.215821 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:22.237459 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:22.262996 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:22.279973 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:22.281605 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:22.281631 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:22.398541 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:22.482149 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:22.499456 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:22.545871 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:22.585240 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:22.630697 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:22.630982 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:22.640527 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:22.642750 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:22.657577 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:22.817754 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:22.857000 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:22.901918 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:22.935786 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:22.940933 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:22.945335 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:22.971246 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:22.973587 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:23.047969 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:23.077899 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:23.083544 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:23.092772 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:23.120456 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:23.192811 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:23.240965 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:23.242579 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:23.338790 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:23.538276 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:23.546818 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:23.578244 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:23.587289 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:23.613742 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:23.812903 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:23.815415 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:23.827445 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:23.864097 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:23.864780 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:23.873992 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:23.899626 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:23.973132 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:23.987396 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:24.002918 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:24.038580 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.062974 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:24.082149 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:24.096051 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.102082 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:24.113209 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.154284 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.159389 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:24.180224 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.228178 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.239706 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 178 similar messages]
W20240502 20:50:24.240923 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.263092 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.352965 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.398367 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:24.416457 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:24.437517 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:24.488024 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:24.511283 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.515079 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:24.543272 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.556928 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:24.585390 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 20:50:24.586287 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.651531 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:50:24.655915 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:24.677403 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:24.707473 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:24.758971 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:24.846571 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 20:50:24.856781 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:24.898402 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:24.949236 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:25.016610 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.017836 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.056560 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:25.062322 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.091879 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:25.186609 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:25.214941 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:25.241380 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:25.246096 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.277937 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.278227 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:25.418962 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:25.456526 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.456861 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:25.460167 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 20:50:25.470708  7013 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:25.470892  7013 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:50:25.471436  7013 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:50:25.471937 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:25.472115 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:50:25.473488 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:25.473676 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:25.473749 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:50:25.473980  7013 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:25.480007 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:25.518833 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:25.523101 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.611227 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.624060 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.627835 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:25.646005 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:25.691367 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:25.718113 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.749982 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:25.877792 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:25.888166 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:25.978508 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:26.033658 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:26.098969 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.269289 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:26.300421 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:50:26.316525 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:26.352180 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:26.426375 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:26.433996 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:26.434237 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:26.482858 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.523077 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:26.528337 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.547258 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:26.565488 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.570400 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.579816 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.584918 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.590147 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.610335 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.657310 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:26.658430 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:26.685431 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.711923 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.775146 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.840394 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:26.856822 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.859100 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:26.891332 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:26.918793 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:26.960307 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:26.968173 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:26.990658 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:27.000880 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:27.056906 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:27.110474 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:27.138986 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:27.210530 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:27.229143 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:27.244422 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:27.351357 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:27.354048 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:27.369408  7529 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:27.369642  7529 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:50:27.370503  7529 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:50:27.371129 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:27.371419 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:50:27.372083 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:50:27.372632 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:27.372754 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:27.372792 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:50:27.372998  7529 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:27.373832 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:27.413856 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:27.421653 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:27.435360 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:27.519564 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:27.547739 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:27.607493 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:27.641014 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:27.684250 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:27.707343 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:27.733526 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:27.735154 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:27.834514 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:27.838503 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:27.877836 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:27.882272 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:27.957846 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:27.958081 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:27.958892 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:28.014918 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:28.036185 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:28.061349 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:28.071573 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:28.173153 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:28.223692 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:28.265105 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:28.335258 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:28.357870 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:28.368139 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:28.497967 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:28.555541 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:28.625782 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:28.707149 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:28.807891 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:28.837312 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:28.850791 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:28.866925 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:28.879124 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:28.915836 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:28.935154 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:28.951815 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:29.019193 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.049486 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.082862 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:29.095995 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.129137 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.145318 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.163492 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.195818 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:29.201298 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:29.228179 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.239503 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.246776 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:50:29.259784 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.270049 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:29.311285 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:29.327447 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:29.332743 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:29.388224 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:29.413719 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.438513 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.532477 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.631444 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:29.672086 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 20:50:29.673197 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:29.676045 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.728282 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:29.766073 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:29.780627 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:29.781016 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:29.863821 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240502 20:50:29.894979 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:50:29.895172 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:29.906142 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:30.001386 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.004280 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.005990 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.057847 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:30.091579 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:30.105294 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.118819 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:30.131582 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:30.181447 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.217343 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:30.253041 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:30.336627 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:30.337122  7945 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:50:30.337267  7945 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:50:30.337723  7945 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:50:30.339015 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:30.339172 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:30.339242 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:50:30.339498  7945 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:30.416646 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:30.422430 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.471129 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.480851 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.495020 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.520367 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:30.529973 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.530593 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.644383 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:30.648907 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:30.666911 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:30.672190 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.676409 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:30.736814 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:30.940071 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:30.950070 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:30.969343 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:30.971995 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:31.030148 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:31.120679 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.134459 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.185439 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:31.214500 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:31.295169 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:31.389055 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:31.458267 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.493943 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:50:31.494937 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.510902 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:31.529551 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:31.548681 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:31.563007 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.564131 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.585574 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.589136 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.591097 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:50:31.610159  8277 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:50:31.610354  8277 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:50:31.611169  8277 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:50:31.611557 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:31.611884 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:50:31.612509 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:31.612579 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:50:31.612785  8277 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:50:31.612823 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:31.637130 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:31.692305 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.716703 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:31.737919 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.769732 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.818811 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.823679 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:31.831616 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.887388 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.907672 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:31.959852 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:31.964942 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:31.965032 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:32.005676 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:32.010594 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:32.125705 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:32.127029 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:32.184351 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:32.192574 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:32.196532 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:32.348423 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:32.356025 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:32.369644 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:32.505712 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:32.523825 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:32.569186 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:50:32.590696 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:32.611213 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:32.612011 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:32.720659 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:32.743244 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:32.781800 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:32.787678 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:32.795497 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:32.832414 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:32.848351 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:32.865211 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:32.888804 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:32.907939 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:33.014580 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:33.019822 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:33.021127 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:33.029162 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:33.029870 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:33.090564 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:33.124764 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:33.208752 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 20:50:33.226050  8465 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:33.226258  8465 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:50:33.226908  8465 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:50:33.227674 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:33.227954 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:50:33.228603 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:33.228768 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:33.228890 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:50:33.229147  8465 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:33.231675 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:33.262559 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:33.359139 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:33.505566 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:33.527274 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:33.538798 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:33.585624 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:33.593959 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:33.768675 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:33.786082 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:33.789245 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:33.802815 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:33.812220 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:33.867954 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:33.947603 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:33.965013 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:33.980456 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:34.016381 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:34.040813 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.070601 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.075804 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.123185 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.130175 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:34.145879 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.147776 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:34.152746 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:34.198169 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:34.309825 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:50:34.310629 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.370149 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.371605 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.438055 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:34.456452 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.463074 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.486390 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.546339 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:34.549407 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:34.563874 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.567183 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:34.575510 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:34.584867 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:34.609136 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.654173 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:34.664918 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:34.665622 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:34.691915 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 20:50:34.827919 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.836694 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:34.843387 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:34.873687 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 20:50:35.017347 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:35.019729 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:35.057507 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:35.059777 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:35.156297 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.225198 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:35.257930 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:50:35.259999 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.262562 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.291222 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.356405 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.399127 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.414315 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:35.451589 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.460264 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:35.513202 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:35.533164 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.561440 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:35.603883 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:35.626231 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:35.661852 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:35.707726  8683 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:35.707978  8683 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:50:35.708820  8683 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:50:35.709398 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:50:35.709594 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:50:35.710227 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:35.710706 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:35.710948 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:50:35.711277  8683 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:35.720511 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:35.739151 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.752684 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.789736 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:35.810660 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:35.846057 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:35.995888 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:36.043637 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:36.051086 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:36.071887 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.085870 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:36.168661 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.225363 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:36.249788 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:36.305341 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:36.391074 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:36.412117 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:36.434217 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.458149 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.466166 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.518157 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:50:36.518920 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.555305 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:36.583703 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:36.613756 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.677608 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:36.702584 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:36.727450 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.742514 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.752058 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.800798 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.863792 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:36.888131 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:36.891708 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:36.903424 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:36.907110 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.937381 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.954908 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.955879 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:36.986181 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:36.987005 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:37.024791 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:37.067631 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:37.096843 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:37.146708 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:37.180541 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:37.308940 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:37.316156 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:37.374426 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:37.374815 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:37.437665 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:37.464140 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:50:37.472191  8683 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:50:37.472438  8683 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:50:37.473384  8683 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
W20240502 20:50:37.474603 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:37.474680 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:50:37.474833 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:37.475605 27298 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:50:37.476548 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:50:37.476974  8683 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:50:37.538384 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:37.583370 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:37.600327 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:37.736407 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:37.758751 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:37.795409 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:37.862013 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:37.862469 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:50:37.920477 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:37.931176 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:50:37.961086  8683 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:50:37.961239  8683 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:50:37.961699  8683 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:50:37.963115 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:37.963488 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:37.963704 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:50:37.963786 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:50:37.964298  8683 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:37.977007 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:38.013949 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:38.039929 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:38.083293 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:38.083335 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:38.111887 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:38.136220 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:38.189303 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:38.231910 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:38.246511 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:38.247676 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:38.281275 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:38.328649 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:38.334337 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:38.377733 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:38.397389 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:38.415675 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:38.479383 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:38.665905 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:38.681391 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:38.719223 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:38.786367 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:38.801239 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:38.806200 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:38.824635 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:38.887590 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:38.928056 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:39.008942 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.151744 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:39.159173 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.161614 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.176052 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.198935 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:39.220786 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:39.249424 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.307574 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:39.320679 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.321187 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:50:39.340159 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.366535 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.370921 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:39.377604 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:39.423846 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.425544 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:39.426719 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.451155 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.491179 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:39.510823 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.510823 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:39.517699 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.574016 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:39.608016 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:39.644529 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.734294 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:39.747148 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:39.759987 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 20:50:39.760901 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:39.761415 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:39.778699 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:39.782172 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:39.811918 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:39.918260 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:39.924151 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:39.927193 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 20:50:40.001173 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:40.042568 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:40.206547 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:40.246197 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:40.313285 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:40.330408 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:40.331125 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:40.423981 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:40.440739 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:40.448484 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:40.455157 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:40.521384 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:50:40.556551 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:40.580021 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:40.587735 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:40.615535 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:40.624897 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:40.625181 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:40.660995 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:40.661029 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:40.728039 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:40.752558 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:40.808665 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:40.823508 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:40.846403 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:40.880761 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:40.893790 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:40.938161 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:41.044531 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:41.046021 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.100286 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:41.124603 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.196566 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:41.202986 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:41.208285 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:41.215967 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.297490 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:41.352514 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:41.454437  6888 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:41.454646  6888 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:50:41.455662  6888 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:50:41.456259 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:50:41.456475 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:50:41.457743 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:41.458189 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:41.458300 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:50:41.458541  6888 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:41.498282 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.543018 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.565544 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.594341 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:50:41.611299 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.635176 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:41.650830 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:41.757848 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:41.766319 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:41.779358 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:41.786499 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:41.795081 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:41.803131 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.820678 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:41.836299 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.882720 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.901113 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.956849 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.960026 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.988401 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:41.998445 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:42.037214 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:42.039897 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:42.131426 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:42.139015 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:42.214974 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:42.236307 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:42.243381 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:42.253955 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:42.323272 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:42.331780 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:42.343179 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:42.370510 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:42.423147 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:42.430337 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:42.548457 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:42.695792 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:42.701004 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:42.706315 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:42.805763 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:42.826491 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:42.881175 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:42.883946 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:42.904182 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:42.922309 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:42.945431 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:42.995281 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:50:43.001569 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:43.051837 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:43.093343 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:43.114500 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:43.172982 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:43.195986 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:43.198993 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:43.207928 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:43.215165 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:43.243839 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:43.257663 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:43.296550 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:43.317032 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:43.367492 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:43.381423 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:43.473520 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:43.529461 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:43.621484 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:43.664412 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:43.713922 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:43.731540 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:43.782812 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:43.787966 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:43.789831 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:43.839960 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:43.903743 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:43.978471 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.075464 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.099334 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:44.172472 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:44.206465 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:44.238400 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.250604 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.321053 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.352757 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:50:44.363775 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:44.371304 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.374542 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:44.382664 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.386808 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:44.396878 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:44.409299 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.424151 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.446082 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:44.470665 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.514065 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.524271 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:44.548465 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:44.648063 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:44.687381 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:44.704560 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.716092 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:44.727212 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:44.736889 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:44.742398 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:44.800254 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 20:50:44.815510 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:44.823833 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:44.867214 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:44.927825 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 20:50:44.991318 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:45.018249 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:45.064254 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:45.111966 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:45.177273 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:45.180263 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:45.220726 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:45.255357 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:45.306015 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:45.453260 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:45.480628 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:45.480800 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:45.495247 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:50:45.507328 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:45.537954 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:45.540705 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:45.581674 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:45.608098 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:45.612015 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:45.630278 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:45.646637 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:45.658882 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:45.682036 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 20:50:45.709383 10655 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:50:45.709571 10655 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:50:45.710111 10655 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:50:45.711815 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:45.711977 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:45.712069 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:50:45.712273 10655 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:45.712805 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:45.777657 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:45.799439 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:45.854497 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:45.901058 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:46.008510 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.014222 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:46.041904 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:46.115216 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:46.121989 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.230595 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.282923 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:46.307271 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.337364 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:46.340865 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:46.411749 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.488696 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:46.519989 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:46.542363 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.556339 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.559136 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.735148 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:46.738452 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:46.779640 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.785605 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.814141 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:46.817641 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:46.867623 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:46.888878 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:50:46.889849 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.907382 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:46.928923 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.940656 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:46.957389 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:46.995860 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:47.015976 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:47.033987 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:47.167212 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:47.168507 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:47.208067 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:47.217422 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:47.245585 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:47.258456 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:47.284199 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:47.298019 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:47.359568 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:47.387986 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:47.403687 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:47.409767 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:50:47.482729 10935 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:50:47.482920 10935 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:50:47.483644 10935 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:50:47.484181 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:47.484336 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:50:47.485206 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:47.485282 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:50:47.485385 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:47.485473 10935 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:50:47.498098 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:47.549994 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:47.654052 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:47.667337 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:47.704159 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:47.770085 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:47.862746 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:47.877228 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:47.932865 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:50:47.946231 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:47.966933 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:48.006997 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:48.030072 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:48.030237 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:48.033664 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:48.047353 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:48.104025 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:48.134997 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:48.197681 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:48.200764 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:48.224962 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:48.238376 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:48.300913 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:50:48.325552 10933 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:48.325721 10933 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:50:48.326228 10933 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:50:48.326817 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:50:48.326994 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:50:48.327642 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:48.327981 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:48.328063 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:50:48.328330 10933 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:48.336524 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:48.343878 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:48.349596 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:48.365235 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:48.394260 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:48.429451 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:48.484526 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:48.555624 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:48.600515 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:48.615803 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:48.790426 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:48.852160 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:48.936005 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:48.937824 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:48.941792 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:48.952212 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:48.954838 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.022383 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:49.035342 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.035884 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:49.116366 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.144809 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:50:49.147258 11123 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:49.147452 11123 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:50:49.148234 11123 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:50:49.148711 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:50:49.148980 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:50:49.150416 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:49.150552 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:49.150614 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:50:49.150815 11123 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:49.174561 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.202863 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:49.352097 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:49.378399 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:49.405151 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240502 20:50:49.406217 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.417140 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.418228 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:49.448400 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:49.450433 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:49.466784 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.508955 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:49.563010 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.617877 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:49.627475 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.637790 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:49.642079 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:49.723546 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.752964 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:49.799405 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:49.823338 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 20:50:49.888639 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:49.926635 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:49.998322 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 20:50:49.999434 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:50.000059 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:50.013244 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:50.039934 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:50.062629 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:50.071591 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:50.079946 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:50.085023 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:50.096300 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:50.144452 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:50.243872 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:50.246003 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:50.397528 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:50.430096 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:50.489183 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:50.542852 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:50.545543 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:50.557579 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:50.565372 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:50.566270 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:50:50.597187 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:50.657276 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:50.666519 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:50.677769 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:50.718775 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:50.731006 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:50.804279 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:50.840903 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:50.876497 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:50.912669 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:50.933817 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:50.972177 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:50.973692 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:50.990932 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:51.067718 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:51.084124 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:51.132469 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:51.148880 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:51.152453 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:51.174247 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:51.303622 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:51.312562 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:51.377439 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:51.461140 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:51.564090 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:51.597319 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:51.597359 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:51.615918 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:50:51.624243 10935 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:50:51.624418 10935 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
W20240502 20:50:51.624852 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:50:51.624866 10935 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:50:51.625982 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:51.626070 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:51.626132 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:50:51.626279 10935 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:51.634950 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:51.743731 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:51.759246 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:51.836805 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:51.840556 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:51.847945 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:51.854317 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:51.927906 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:51.953361 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:51.976564 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:52.029214 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 20:50:52.030010 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:52.076889 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:52.109772 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:52.120594 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:52.153991 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:52.162144 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:52.197007 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:52.218174 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:52.237788 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:52.389158 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:52.426463 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:52.454615 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:52.510138 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:52.522612 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:52.524838 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:52.571164 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:52.621138 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:52.632902 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:52.634712 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:52.688753 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:52.702772 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:52.713691 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:52.762488 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:52.837352 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:52.863399 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:52.909581 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:52.933633 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:52.966586 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:52.982215 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:50:52.990574 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:53.036329 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:53.071846 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:53.181674 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:53.181684 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:53.188585 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:53.191109 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:53.223099 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:53.227463 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:53.328734 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:53.343387 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:53.353324 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:53.369249 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:53.378551 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:53.401178 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:53.446733 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:53.497498 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:53.549144 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:53.564769 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:53.625165 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:53.669015 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:53.680673 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:53.712085 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:53.904934 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:53.987576 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.028076 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.066740 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:54.067607 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:54.074229 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.098403 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:54.108135 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:54.109354 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.116041 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:54.269320 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.273156 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.296972 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:54.299925 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:54.328382 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:54.340632 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.359856 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.424769 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 20:50:54.443712 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.549718 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:54.553727 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.585559 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:54.590360 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.632292 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:54.719020 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:54.721979 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.743645 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:54.758687 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:54.791755 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:54.816758 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:54.825420 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240502 20:50:54.881855 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.898566 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:54.900993 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:54.966780 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:55.063477 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 20:50:55.099229 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:55.102900 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:55.103333 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:55.138919 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:55.142567 11725 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:50:55.142764 11725 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:50:55.143420 11725 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:50:55.143983 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:50:55.144163 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:50:55.145320 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:55.145450 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:50:55.145515 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:50:55.145851 11725 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:50:55.181833 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:55.205453 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:55.214849 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:55.241165 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:55.247212 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:55.250736 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:55.282414 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:50:55.297317 12139 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:50:55.301813 12139 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:50:55.302857 12139 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:50:55.303635 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:50:55.303704 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:50:55.304756 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:50:55.305039 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:50:55.305317 12139 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:50:55.305742 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:55.379215 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:55.401029 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:55.406098 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:55.491613 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:55.519470 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:55.540318 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:55.592562 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:55.594743 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:55.693673 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:55.704929 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:55.742630 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:55.744750 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:55.746533 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:55.801297 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:55.807899 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:55.808380 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:55.820860 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:50:55.923252 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:55.974869 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:55.994223 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:56.058559 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:56.059872 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:56.125530 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:56.176816 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:56.206332 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:56.237529 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:56.265738 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:56.404490 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:56.409442 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:56.463754 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:56.488162 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:56.514402 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:56.516291 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:56.601684 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:56.612462 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:56.619710 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:56.629721 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:56.695328 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:56.707486 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:56.746497 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:56.806102 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:56.944665 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:56.957510 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:56.979517 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:56.998836 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:57.002977 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:57.049949 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:50:57.132123 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:57.151005 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:57.159502 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:57.195991 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:57.224411 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:57.232949 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:57.298036 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:57.340761 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:57.391132 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:57.403262 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:57.449535 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:57.473380 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:57.473991 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:57.483454 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:57.486380 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:57.487463 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:57.511515 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:57.526561 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:57.568387 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:57.574995 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:57.601624 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:57.652439 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:57.799772 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:57.822873 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:57.843936 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:57.944407 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:57.954169 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:57.957778 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:57.975025 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:50:58.002597 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:58.032665 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:58.083765 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:58.200924 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:58.205941 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:58.222538 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:50:58.227288 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:58.234756 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:58.240383 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:58.270653 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:50:58.359447 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:58.404085 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:58.430475 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:58.461259 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:58.472834 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:58.473793 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:58.604173 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:58.613714 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:50:58.635918 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:58.651477 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:58.685900 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:58.731343 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:58.783924 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:58.873282 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:58.939697 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:58.942569 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:58.984887 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:59.015424 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:50:59.023643 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:50:59.054667 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:59.092459 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:59.114298 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:59.140082 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:59.145488 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:59.195847 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:59.261513 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:59.264935 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:59.457099 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:59.472546 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240502 20:50:59.522387 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:50:59.532917 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:50:59.540028 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:59.558907 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:59.659286 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:59.664795 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:59.730485 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:50:59.764215 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:59.789394 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:59.830610 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240502 20:50:59.831360 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:50:59.863034 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:59.891806 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:50:59.906009 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:50:59.919111 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:50:59.974795 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:50:59.978065 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:00.017592 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:00.030864 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:00.053987 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:00.110817 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:00.152566 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 20:51:00.154431 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:00.164419 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:00.222262 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:00.251619 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:00.266022 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:00.270900 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:00.278946 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:00.301607 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:51:00.328447 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:00.349632 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:00.365468 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:00.401811 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:00.473089 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:00.541921 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:00.549794 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:00.565891 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:00.585801 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:00.640376 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:00.690690 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:51:00.704818 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:00.762408 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:00.812960 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:00.830500 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:00.833559 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:00.859014 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:00.943702 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:00.944768 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:00.946744 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:01.004833 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:01.089286 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:01.107336 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:01.111248 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:01.188838 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:01.195533 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:01.242163 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:01.256989 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:01.295938 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:01.367174 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:01.368491 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:01.375782 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:01.414176 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:01.433710 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:01.502235 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:01.591126 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:01.703820 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:01.708858 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:01.715029 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:01.764745 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:01.813396 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:01.952813 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:01.990291 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:02.020133 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:02.023787 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:02.102594 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:51:02.139472 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:02.198032 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:02.230100 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:02.237354 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:02.246099 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:02.262756 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:02.337551 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:02.354679 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:02.374696 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:02.374822 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:02.382565 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:02.385670 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:02.436950 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:02.449836 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:02.518107 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:02.553527 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:02.657924 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:02.679337 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:02.684458 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:02.694941 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 20:51:02.740671 13348 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:51:02.740934 13348 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:02.741806 13348 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:51:02.742318 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:02.742892 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:51:02.743763 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:51:02.743899 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:51:02.744071 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:02.744189 13348 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:51:02.747121 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:02.747699 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:02.773528 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:02.786356 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:02.830922 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:02.833923 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:02.852349 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:02.867070 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:02.996053 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:03.016255 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:03.079082 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:03.088232 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:03.093555 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:03.212105 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:03.225739 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:03.283732 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:03.288420 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:51:03.301995 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:03.313637 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:03.313918 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:03.315057 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:03.316679 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:03.417619 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:03.417804 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 20:51:03.443392 13596 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:03.443593 13596 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:51:03.444360 13596 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:51:03.444758 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:03.444952 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:51:03.446163 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:03.446316 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:03.446358 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:51:03.446542 13596 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:03.463907 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:03.580142 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:03.611882 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:03.667922 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:03.676368 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:03.677367 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:03.692763 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:03.696642 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:03.724859 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:03.725703 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:03.758364 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:03.810068 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:03.818887 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:03.830909 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:03.860445 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:03.866768 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:03.914343 13596 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:51:03.914515 13596 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:51:03.915225 13596 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:51:03.917217 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:03.917459 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:51:03.917549 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:51:03.917989 13596 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:03.976186 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.139503 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.243664 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:04.246227 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.267320 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:04.270258 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.272753 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:04.273252 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:04.311234 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:04.435523 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:04.442368 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:04.454612 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.472597 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 182 similar messages]
I20240502 20:51:04.511222 13596 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:04.511402 13596 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:51:04.511965 13596 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:51:04.512738 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:51:04.512951 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:51:04.513896 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:04.514034 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:51:04.514135 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:51:04.514289 13596 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:04.563432 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.580775 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:04.663074 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.670387 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:04.705183 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:04.722710 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:04.746153 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.784066 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:04.793867 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.824031 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:04.843921 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:04.859359 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 20:51:04.867434 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.894155 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:04.923696 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:04.973850 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:05.006170 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:05.043314 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:05.059108 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:05.188776 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:05.236431 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:05.244880 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 20:51:05.294056 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:05.300173 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:05.339843 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:05.348960 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:05.352890 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:05.420168 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:05.428089 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:05.484632 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:05.497438 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:05.550945 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:05.570418 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:05.643987 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:05.689982 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:05.741964 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:05.758376 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:05.787972 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:05.805909 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:05.826476 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:05.850116 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:05.862501 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:05.905440 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:05.931756 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:05.931756 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:05.955338 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:05.994374 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:06.005559 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:06.038753 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:51:06.047472 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:06.048312 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:06.087543 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:06.220297 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:06.308825 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:06.312741 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:06.386914 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:06.389348 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:06.397426 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:06.398963 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:06.399056 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:06.405687 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:06.453872 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:06.492511 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:06.579016 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:06.613528 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:06.660122 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:06.717840 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:06.816342 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:06.850872 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:06.872749 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:06.899866 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:06.951146 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:07.004776 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:07.064294 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:07.092460 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:07.098702 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:07.121656 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 20:51:07.121706 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:07.123805 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:07.208685 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:07.275065 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:07.358039 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:07.372493 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:07.415813 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:07.427589 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:07.428682 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:07.478166 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:07.480958 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:07.509335 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:07.573431 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:07.625947 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:07.700079 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:07.711221 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:07.737499 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:07.757964 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:07.802101 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:07.850416 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:07.885655 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:07.946729 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:07.957911 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:08.015578 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:08.017091 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:08.051760 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:08.060397 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:08.083390 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:08.102586 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:08.124619 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:08.166817 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:08.211514 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:08.301460 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:08.322678 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:08.352730 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:08.356536 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:08.411151 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:08.431478 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:08.439312 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:08.592360 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:08.593487 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:51:08.594831 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:08.596876 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:08.635558 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:08.650297 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:08.653425 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:08.679658 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:08.688235 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:08.699307 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:08.763236 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:08.801705 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:08.815913 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:08.832443 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:08.880708 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:08.881201 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:08.941267 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:08.980152 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:08.998641 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:09.009699 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:09.050105 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:09.170040 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:09.210847 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:09.230485 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:09.239107 14593 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:09.239338 14593 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:51:09.240094 14593 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:51:09.240862 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:51:09.241147 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:51:09.241674 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:09.242022 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:51:09.242120 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:51:09.242329 14593 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:09.330682 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:09.343447 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:09.486150 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:09.499739 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:09.521844 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240502 20:51:09.522567 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:09.525523 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:09.540874 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:09.637181 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:09.684001 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:09.711397 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:09.713573 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:09.714138 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:09.754829 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:09.755246 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:51:09.784180 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:09.862250 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240502 20:51:09.931874 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:09.965651 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:10.011225 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:10.106112 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:10.125032 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:10.159564 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:10.162667 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:10.201555 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:10.226038 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:10.248661 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:10.269801 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240502 20:51:10.290699 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:10.301012 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:10.373924 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:10.397326 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:10.426617 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:10.436304 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:10.451769 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:10.490563 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:10.490649 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:10.543830 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:10.573964 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:10.674737 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:10.684346 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:10.701871 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:10.708434 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:10.771580 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:10.819996 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:10.835608 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:51:10.850008 14442 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:51:10.850225 14442 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:51:10.850844 14442 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:51:10.852414 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:10.852552 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:51:10.852599 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:51:10.852787 14442 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:10.855388 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:10.874433 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:10.902807 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:10.921427 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 20:51:10.922211 14442 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:51:10.922442 14442 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:10.923141 14442 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:51:10.923616 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:10.923905 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:51:10.924434 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:10.924656 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:51:10.924743 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:51:10.924968 14442 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:51:10.957155 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:10.971307 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:11.021725 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.079592 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:11.082718 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:11.133009 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:11.176417 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.187132 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:11.245424 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:11.267841 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.284121 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.287184 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.291180 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:11.298197 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:51:11.344199 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:11.364933 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.366495 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:11.511317 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.516377 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.628600 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:11.671077 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:11.690397 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:11.733779 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.755501 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:11.783869 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:11.813983 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.871459 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:11.932344 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:11.944566 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:12.064524 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:12.080503 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:12.109688 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:12.144080 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:12.194658 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:12.234331 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:12.234550 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:12.247324 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:51:12.250984 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:12.352540 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:12.472595 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:12.487550 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:12.555238 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:12.570784 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:12.623203 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:12.632427 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:12.632434 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:12.650398 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:12.684849 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:12.706671 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:12.739773 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:12.763504 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:12.792218 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:12.800846 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:12.825649 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:12.827972 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:12.899394 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:12.913587 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:12.918285 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:12.925813 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:12.967326 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:13.019968 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:13.033860 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:13.040853 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:13.052755 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:13.189329 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:13.230749 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:13.241667 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:13.249738 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:13.249852 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:13.258263 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:13.283857 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:13.297207 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:13.339812 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:51:13.350903 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:13.405256 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:13.406581 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:13.435249 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:13.491238 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:13.512269 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:13.567695 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:13.607489 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:13.623003 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:13.635762 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:13.672389 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:13.674320 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:13.686383 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:13.894122 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:13.894198 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:13.986284 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:14.011303 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:14.061237 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:14.089754 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:14.093158 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:14.131906 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:14.135028 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:14.157862 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:14.176127 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:14.353956 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:14.387022 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:14.415953 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:14.532186 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 183 similar messages]
W20240502 20:51:14.533113 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:14.563594 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:14.582747 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:14.593405 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:14.618846 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:14.666934 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:14.824030 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:14.865556 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:14.867202 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 20:51:14.873407 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:14.917304 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:14.917896 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:14.940972 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:14.948099 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:15.021214 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:15.067715 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.101054 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.197809 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.208557 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:15.221218 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:15.242856 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.258355 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.279671 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 20:51:15.299093 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:15.326687 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.350672 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:15.399742 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:15.409940 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:15.410996 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 20:51:15.419943 15654 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:15.420142 15654 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:51:15.420704 15654 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:51:15.421288 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:51:15.421478 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:51:15.422283 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:15.422401 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:15.422497 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:51:15.422755 15654 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:15.427935 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:15.431143 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:15.449669 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.492297 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:15.500232 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:15.517314 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:15.526718 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:15.564074 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:15.614550 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:15.634522 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:15.657682 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:15.669461 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.676466 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:51:15.691255 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:15.714073 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.746282 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.784253 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:15.795195 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:15.797732 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:15.832954 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:15.843878 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:15.846550 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:15.930963 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:16.102758 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:16.116029 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:16.192795 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:16.202672 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:16.218040 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:16.223387 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:16.268919 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:16.324775 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:16.337304 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:16.378696 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:16.439654 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:16.528692 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:16.569389 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:16.585161 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:16.589143 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:16.625568 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:16.678118 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:16.683176 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:16.722030 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:16.842111 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:17.005414 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.029771 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:17.064639 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.072764 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.117556 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:17.119473 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:17.168850 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:17.249215 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:17.258299 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:17.307427 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:51:17.308213 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.398070 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.423131 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.427567 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:17.427646 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:17.443456 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:17.487097 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:17.518548 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.574023 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:17.595225 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:17.636790 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:17.669651 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:17.704605 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:17.705291 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.712463 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.751564 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:17.816494 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:17.852952 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.882472 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.938901 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:17.959995 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:17.982035 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:17.987884 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:18.006757 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.016386 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:18.016736 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:18.021426 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.050184 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.074942 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:18.122900 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:18.131300 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:51:18.133180 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.138949 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.148716 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:18.162214 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:18.169601 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.192734 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.215461 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:18.249495 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:18.298825 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:18.338452 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:18.437549 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:18.437645 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:18.455230 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:18.516089 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:18.522471 16117 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:51:18.522632 16117 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:51:18.523149 16117 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:51:18.524506 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:18.524902 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:18.524982 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:51:18.525166 16117 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:18.674099 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:18.733959 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.734586 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.827127 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.843314 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:18.849937 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:18.862809 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:18.899044 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:18.902269 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:18.950189 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:18.960722 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:19.045591 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:19.059484 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:19.103106 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:19.201428 16117 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:19.201637 16117 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:51:19.202312 16117 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:51:19.202915 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:19.203289 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:51:19.203953 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:19.204053 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:19.204088 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:51:19.204241 16117 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:19.244131 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:19.317620 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:19.345947 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:19.440721 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:19.492319 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:19.522351 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:19.542260 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 178 similar messages]
W20240502 20:51:19.591317 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:19.673700 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:19.675631 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:19.705233 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:19.716061 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:19.788192 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:19.801342 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:19.812196 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:19.855064 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:19.862810 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:51:19.867777 16022 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:19.867947 16022 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
W20240502 20:51:19.868536 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 154 similar messages]
I20240502 20:51:19.868860 16022 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:51:19.869630 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:51:19.869820 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:51:19.870266 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:19.870355 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:51:19.870450 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:51:19.870607 16022 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:19.874814 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:19.886391 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:19.899089 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:19.966744 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:20.113867 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:20.149339 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:20.153483 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:20.251716 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:20.293845 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:20.294742 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:20.301458 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 20:51:20.303186 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:20.326059 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:20.386998 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:20.388806 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:20.416527 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:20.453505 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:20.458606 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:20.459470 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:20.467571 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:20.505916 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:20.533166 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:20.540458 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:20.577530 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:20.577607 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:20.589279 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:20.605124 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:51:20.638279 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:20.689595 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:20.695262 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:20.735283 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:20.739065 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:20.745326 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:20.757498 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:20.768239 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:20.771093 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:20.787509 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:20.793396 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:20.844116 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:20.892789 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:20.926381 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:21.204557 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:21.209553 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:21.237656 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:21.289569 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:21.334342 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:21.362085 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:21.386216 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:21.426846 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:21.431913 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:21.526613 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:21.548377 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:21.548573 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:21.629562 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:21.717615 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:21.797175 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:21.804767 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:21.857369 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:21.864594 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:21.964768 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:21.974711 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:21.976313 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:22.058116 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:22.154242 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:22.154881 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:22.175494 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:22.208328 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:22.287670 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:22.301287 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:22.317816 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:22.386960 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:22.390082 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:22.409350 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 20:51:22.437140 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:22.443102 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:22.521209 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:22.525135 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:22.545235 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:22.588294 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:22.735070 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:22.761538 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:22.774116 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:22.777535 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:22.848323 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:22.851616 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:51:22.880076 16819 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:22.880337 16819 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:51:22.881183 16819 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:51:22.881891 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:22.882149 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:51:22.882928 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:22.883081 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:22.883141 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:51:22.883327 16819 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:22.884315 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:22.887387 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:22.905269 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:22.917800 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:22.922833 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:22.932390 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:22.951582 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:22.979840 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:22.987111 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:22.992798 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:22.999475 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:23.037546 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:23.042497 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:51:23.067569 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:23.094146 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:23.120868 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:23.201000 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:23.216439 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:23.255152 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:23.279475 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:23.295056 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:23.341553 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:23.409296 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:23.410045 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:23.442965 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:23.488446 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:23.527817 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:23.533608 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:23.642458 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:23.860018 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:23.892678 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:23.908221 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:24.021487 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.024132 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.026767 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:24.047622 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.084246 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:24.104218 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.116614 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:24.124574 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.138695 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:24.321484 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.326175 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.373976 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:24.382513 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:24.425243 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:24.471028 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:24.503319 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:24.537285 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.609856 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240502 20:51:24.638993 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.647326 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.679137 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:24.746419 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.784575 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:24.825430 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:24.895396 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 20:51:24.898535 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:24.910583 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:24.912456 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:24.950502 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:24.972927 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:24.996292 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:25.013801 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:25.022673 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.058879 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.098881 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.186954 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:25.215376 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.242844 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:25.245862 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:25.274758 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.318349 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.334349 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 20:51:25.335927 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:25.381026 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:25.381213 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:25.420212 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:25.422263 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.428615 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:25.429370 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:25.480500 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:25.483392 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.488554 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:25.489748 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:25.547348 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.584178 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:51:25.629777 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.650760 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:25.715201 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:25.718597 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.811770 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:25.824270 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:25.869838 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:25.880606 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.884196 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:25.943089 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:25.978652 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:51:25.998435 17328 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:25.998626 17328 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
W20240502 20:51:25.998813 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 20:51:25.999572 17328 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:51:25.999893 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:51:26.000104 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:51:26.001650 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:26.001806 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:26.001864 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:51:26.002033 17328 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:26.042924 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:26.067071 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:26.082067 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:26.107885 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:26.269218 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:26.274350 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:26.291132 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:26.423450 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:26.465198 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:26.472649 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:26.504643 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:26.534934 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:26.567951 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:26.593462 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:26.642218 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:26.668403 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:26.674031 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:26.696509 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:26.881538 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:26.894212 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:26.898730 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:26.918442 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:27.005986 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:27.023648 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:27.032832 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:27.092157 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:27.094012 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:27.142403 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:27.388650 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:27.445468 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:27.470989 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:27.493249 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:27.497349 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 20:51:27.498354 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:27.515426 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:27.528575 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:27.532153 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:27.533843 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:27.540520 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:27.550632 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:27.579648 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:27.633481 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:27.673333 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:27.704586 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:27.722908 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:27.726581 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:27.730111 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:27.830363 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:27.868227 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:27.882768 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:27.885017 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:27.946622 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:27.972739 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:27.993608 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:28.025355 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:28.035151 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:28.039916 17519 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:28.040158 17519 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:51:28.040896 17519 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:51:28.041644 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:28.041870 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:51:28.042376 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:28.042537 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:28.042596 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:51:28.042893 17519 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:28.069532 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:28.095369 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:28.095788 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:28.106290 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:28.110062 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:28.112641 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:28.133968 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:28.162837 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:51:28.212457 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:28.231875 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:28.288347 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:28.463352 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:28.470471 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:51:28.493846 17519 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:51:28.494062 17519 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:28.494812 17519 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:51:28.495467 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:51:28.496109 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:28.496738 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:28.497483 27298 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:51:28.497772 17519 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:51:28.497792 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:28.500475 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:28.503827 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:28.526398 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:28.557885 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:28.570964 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:28.579967 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:28.585806 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:28.586634 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:28.667727 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:28.678200 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:28.847836 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:51:28.900101 17118 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:28.900352 17118 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:51:28.901245 17118 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:51:28.902037 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:51:28.902343 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:51:28.902945 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:28.903292 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:28.903371 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:51:28.903568 17118 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:28.904789 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:28.960494 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:28.990263 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:28.998262 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:29.004103 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:29.015062 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:29.087141 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:29.096966 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:29.098904 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:29.225678 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:29.296851 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:29.371829 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:29.395581 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:29.446333 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:29.542140 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:29.551254 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:29.608184 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:29.630924 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:29.638746 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:29.640707 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240502 20:51:29.641803 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:29.785454 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:29.849010 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:29.884068 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:29.902292 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 20:51:29.950533 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:29.965718 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:29.998392 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.063639 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.092263 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:30.122553 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:30.125105 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:30.127641 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:30.130837 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:30.155452 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:30.186748 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.216158 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:30.227519 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.241207 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.254686 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:30.337258 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240502 20:51:30.338222 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:30.419649 27158 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:30.464579 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:30.474372 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:30.509303 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:30.513651 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:30.530989 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.530895 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.533130 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.603726 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.619132 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:30.620754 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:30.678987 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.693310 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.713583 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:30.720867 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:51:30.727273 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:30.727633 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:30.736109 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:30.826720 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:30.903026 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:30.948710 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:30.976974 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:30.994199 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:31.048568 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:31.051841 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:31.059479 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:31.090750 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:31.115520 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:31.120325 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:31.169127 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:31.248457 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:51:31.315299 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:31.364638 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:31.364876 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:31.388566 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:31.438148 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:31.453104 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:31.475102 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:31.546736 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:31.736248 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:31.740797 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:31.754184 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:31.825896 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:31.826921 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:31.851053 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:51:31.873776 17519 consensus_queue.cc:579] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.233s)
W20240502 20:51:31.875521 27295 consensus_peers.cc:462] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:51:31.881842 27361 consensus_queue.cc:237] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:51:31.884289 27227 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 20:51:31.884837 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:31.884886 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:31.885145 18502 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:31.887833 18503 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:51:31.888496 27227 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:51:31.890551 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f00d12e425284c36ae8baed56c7cfaf6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:31.891391 26807 catalog_manager.cc:5484] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 8 to 9, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
W20240502 20:51:31.898054 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:31.899086 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:51:32.043010 18271 consensus_queue.cc:579] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.247s)
W20240502 20:51:32.044545 27158 consensus_peers.cc:462] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:51:32.050961 27227 consensus_queue.cc:237] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
W20240502 20:51:32.051541 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:51:32.053746 27361 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:51:32.053807 26957 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:51:32.054415 18519 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:51:32.054610 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.054790 18271 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:51:32.057494 18519 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:51:32.058036 26957 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:51:32.059986 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd1b1316da2d47938164ac5b75f11bae with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:32.061012 26807 catalog_manager.cc:5484] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:51:32.061300 27361 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
W20240502 20:51:32.109470 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:32.143015 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:51:32.220613 18566 consensus_queue.cc:579] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.113s)
W20240502 20:51:32.222698 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:32.267267 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:32.293020 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:32.293957 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:51:32.316290 18502 consensus_queue.cc:579] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.096s)
I20240502 20:51:32.319552 27361 consensus_queue.cc:237] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } }
I20240502 20:51:32.321354 27090 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:51:32.321491 26960 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:51:32.321856 18503 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 20:51:32.322192 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.322750 18503 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:51:32.324963 17519 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:51:32.325867 26960 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:51:32.327150 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40df15a82a624913b3b712687d999134 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 20:51:32.327682 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:51:32.328038 27090 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } }
I20240502 20:51:32.329273 26805 catalog_manager.cc:5484] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 5 to 6, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:32.333272 27361 consensus_queue.cc:237] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:51:32.337031 26960 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:51:32.337540 27090 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:51:32.337903 27298 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): Couldn't send request to peer c107c43e538d48f58625c4320f0aa4b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40df15a82a624913b3b712687d999134. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.338169 18581 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:51:32.339080 18581 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:51:32.339744 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.340421 17519 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:51:32.343119 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 40df15a82a624913b3b712687d999134 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:32.343637 27090 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:51:32.344707 26807 catalog_manager.cc:5484] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 6 to 7, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:32.346824 26960 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:51:32.454239 18581 consensus_queue.cc:579] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.018s)
I20240502 20:51:32.457329 27361 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:51:32.458808 27227 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:51:32.459323 18587 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:51:32.459515 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.461503 18581 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:51:32.462181 27227 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:51:32.463754 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5cf737e20ca74db1b94ccff5d60133ee with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:32.465484 26805 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 9 to 10, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:32.470496 27361 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
W20240502 20:51:32.472652 27295 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cf737e20ca74db1b94ccff5d60133ee. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.472663 27227 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:32.473165 18587 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:51:32.473328 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.475190 18581 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:32.475929 27227 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:32.477368 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5cf737e20ca74db1b94ccff5d60133ee with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:32.479863 26807 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 10 to 11, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:32.535123 17470 consensus_queue.cc:579] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 for more than 300 seconds (300.407s)
I20240502 20:51:32.538477 27090 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
W20240502 20:51:32.538715 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 20:51:32.539420 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:32.540521 27027 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): Couldn't send request to peer c107c43e538d48f58625c4320f0aa4b8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.540544 26960 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:32.541186 18567 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:51:32.541450 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.544479 18566 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:51:32.545313 26960 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:51:32.548050 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fdd2311ec0b645f5a79098a32c6deba2 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:32.549417 26805 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 10 to 11, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:32.560370 18622 consensus_queue.cc:579] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.329s)
W20240502 20:51:32.596217 27295 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 20:51:32.602406 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:51:32.644943 18526 consensus_queue.cc:579] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.400s)
I20240502 20:51:32.649770 27227 consensus_queue.cc:237] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } }
I20240502 20:51:32.652732 27090 raft_consensus.cc:1261] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:51:32.652835 26960 raft_consensus.cc:1261] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 20:51:32.654292 18519 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:51:32.654986 18643 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 20:51:32.659511 18526 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } }
I20240502 20:51:32.660352 18567 consensus_queue.cc:579] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 for more than 300 seconds (300.330s)
I20240502 20:51:32.662197 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:32.662976 26805 catalog_manager.cc:5484] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } }
I20240502 20:51:32.663048 27090 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } }
I20240502 20:51:32.664280 26960 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } }
I20240502 20:51:32.669226 27227 consensus_queue.cc:237] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:51:32.671479 26960 raft_consensus.cc:1261] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 20:51:32.672755 27160 consensus_peers.cc:485] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 -> Peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): Couldn't send request to peer ebd8300729ec4226acc38b9cff2d8d3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62c71a33fbcc4cfdbf0065a3fc9dfeba. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.672814 27090 raft_consensus.cc:1261] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 20:51:32.673224 18643 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:51:32.673533 18519 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 20:51:32.677058 18519 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:51:32.677524 17118 consensus_queue.cc:579] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Leader has been unable to successfully communicate with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 for more than 300 seconds (300.108s)
I20240502 20:51:32.678489 26960 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:51:32.679545 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:32.680545 27090 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:51:32.681483 18526 consensus_queue.cc:579] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 for more than 300 seconds (300.263s)
I20240502 20:51:32.682909 26807 catalog_manager.cc:5484] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 5 to 6, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:32.691816 26960 consensus_queue.cc:237] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:51:32.694466 27227 consensus_queue.cc:237] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
W20240502 20:51:32.696192 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:32.696473 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.697839 27090 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:51:32.698740 18526 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:51:32.699935 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1545142b0bd40859bf33bf8686002ae. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:32.700845 26894 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): Couldn't send request to peer c107c43e538d48f58625c4320f0aa4b8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.701862 27361 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:32.702353 17118 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 20:51:32.704493 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.704612 18526 consensus_queue.cc:579] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 for more than 300 seconds (300.261s)
I20240502 20:51:32.714151 18519 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:32.715131 18661 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:51:32.718675 27361 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:51:32.719218 27090 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:32.719564 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f1545142b0bd40859bf33bf8686002ae with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:32.722656 26794 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a2b61909194d488682afd91c0f796405 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:32.724341 26805 catalog_manager.cc:5484] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 6 to 7, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:32.725776 26807 catalog_manager.cc:5484] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 7 to 8, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
W20240502 20:51:32.738827 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:32.741142 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:32.742274 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:32.772287 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:32.806505 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:32.815555 18587 consensus_queue.cc:579] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 for more than 300 seconds (300.497s)
I20240502 20:51:32.818993 27361 consensus_queue.cc:237] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:32.820881 27227 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:51:32.821336 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.821384 18587 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:51:32.821841 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:32.824148 18581 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:32.826073 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 843a26fc48844bfd82911b6a27833023 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:32.827255 27227 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:32.827680 26806 catalog_manager.cc:5484] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 6 to 7, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:32.860580 18686 ts_tablet_manager.cc:902] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:51:32.861378 18686 tablet_copy_client.cc:308] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:51:32.862193 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 40df15a82a624913b3b712687d999134 from peer c107c43e538d48f58625c4320f0aa4b8 ({username='slave'} at 127.19.66.131:37981)
I20240502 20:51:32.862311 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 40df15a82a624913b3b712687d999134 from peer c107c43e538d48f58625c4320f0aa4b8 at {username='slave'} at 127.19.66.131:37981: session id = c107c43e538d48f58625c4320f0aa4b8-40df15a82a624913b3b712687d999134
I20240502 20:51:32.863683 27382 tablet_copy_source_session.cc:205] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:32.865696 18686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40df15a82a624913b3b712687d999134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:32.868366 18686 tablet_copy_client.cc:783] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:32.876616 18686 tablet_copy_client.cc:647] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:32.878751 18686 tablet_copy_client.cc:523] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:32.881390 18686 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:51:32.891294 18686 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:32.891664 18686 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap complete.
I20240502 20:51:32.891886 18686 ts_tablet_manager.cc:1367] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.001s
I20240502 20:51:32.892419 18686 raft_consensus.cc:348] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:51:32.892554 18686 raft_consensus.cc:729] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: LEARNER
I20240502 20:51:32.892668 18686 consensus_queue.cc:260] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:51:32.893100 18686 ts_tablet_manager.cc:1398] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:51:32.893901 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session c107c43e538d48f58625c4320f0aa4b8-40df15a82a624913b3b712687d999134 received from {username='slave'} at 127.19.66.131:37981
I20240502 20:51:32.894011 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session c107c43e538d48f58625c4320f0aa4b8-40df15a82a624913b3b712687d999134 on tablet 40df15a82a624913b3b712687d999134 with peer c107c43e538d48f58625c4320f0aa4b8
W20240502 20:51:32.895058 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:32.924338 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:32.933969 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:32.945327 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 40df15a82a624913b3b712687d999134: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:32.954607 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:32.972548 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:32.988938 18729 ts_tablet_manager.cc:902] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:51:32.989761 18729 tablet_copy_client.cc:308] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:51:32.991207 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 5cf737e20ca74db1b94ccff5d60133ee from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:46731)
I20240502 20:51:32.991461 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 5cf737e20ca74db1b94ccff5d60133ee from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:46731: session id = 297de823852d4a8da44791ab03131cd3-5cf737e20ca74db1b94ccff5d60133ee
I20240502 20:51:32.992985 27382 tablet_copy_source_session.cc:205] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:32.994405 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf737e20ca74db1b94ccff5d60133ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:32.994918 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(40df15a82a624913b3b712687d999134): 4586 bytes on disk
I20240502 20:51:32.996469 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(40df15a82a624913b3b712687d999134) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 20:51:32.997588 18729 tablet_copy_client.cc:783] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 5 data blocks...
W20240502 20:51:32.998073 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:51:33.004343 18729 tablet_copy_client.cc:647] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:33.006237 18729 tablet_copy_client.cc:523] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:33.009020 18729 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:51:33.020462 18729 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:33.020864 18729 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:51:33.021154 18729 ts_tablet_manager.cc:1367] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.012s	user 0.011s	sys 0.003s
I20240502 20:51:33.021652 18729 raft_consensus.cc:348] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:33.021807 18729 raft_consensus.cc:729] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:51:33.021914 18729 consensus_queue.cc:260] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:33.022302 18729 ts_tablet_manager.cc:1398] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:51:33.022866 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-5cf737e20ca74db1b94ccff5d60133ee received from {username='slave'} at 127.19.66.129:46731
I20240502 20:51:33.022949 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 297de823852d4a8da44791ab03131cd3-5cf737e20ca74db1b94ccff5d60133ee on tablet 5cf737e20ca74db1b94ccff5d60133ee with peer 297de823852d4a8da44791ab03131cd3
W20240502 20:51:33.067114 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 20:51:33.079612 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:33.085456 18587 consensus_queue.cc:579] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.476s)
I20240502 20:51:33.089387 27361 consensus_queue.cc:237] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:51:33.091065 26960 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:51:33.091724 18503 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:51:33.091882 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:33.092095 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.093926 17519 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:51:33.094477 26960 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } }
I20240502 20:51:33.096383 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a8ce1df86914b2caf2d083d97eda4e6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:33.097139 26805 catalog_manager.cc:5484] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 9 to 10, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:33.101707 18686 ts_tablet_manager.cc:902] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Initiating tablet copy from peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:51:33.102267 18686 tablet_copy_client.cc:308] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:38835
I20240502 20:51:33.102865 26980 tablet_copy_service.cc:140] P 297de823852d4a8da44791ab03131cd3: Received BeginTabletCopySession request for tablet a2b61909194d488682afd91c0f796405 from peer c107c43e538d48f58625c4320f0aa4b8 ({username='slave'} at 127.19.66.131:58105)
I20240502 20:51:33.103042 26980 tablet_copy_service.cc:161] P 297de823852d4a8da44791ab03131cd3: Beginning new tablet copy session on tablet a2b61909194d488682afd91c0f796405 from peer c107c43e538d48f58625c4320f0aa4b8 at {username='slave'} at 127.19.66.131:58105: session id = c107c43e538d48f58625c4320f0aa4b8-a2b61909194d488682afd91c0f796405
I20240502 20:51:33.104262 26980 tablet_copy_source_session.cc:205] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:33.105875 18686 ts_tablet_manager.cc:1875] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:33.108661 18686 ts_tablet_manager.cc:1888] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:51:33.108945 18686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b61909194d488682afd91c0f796405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:33.110558 18686 tablet_copy_client.cc:783] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:33.117887 18686 tablet_copy_client.cc:647] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:33.119975 18686 tablet_copy_client.cc:523] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:33.121994 18686 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:51:33.130623 18686 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:33.130973 18686 tablet_bootstrap.cc:492] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap complete.
I20240502 20:51:33.131220 18686 ts_tablet_manager.cc:1367] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240502 20:51:33.131682 18686 raft_consensus.cc:348] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:51:33.131846 18686 raft_consensus.cc:729] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: LEARNER
I20240502 20:51:33.131978 18686 consensus_queue.cc:260] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:51:33.132248 18686 ts_tablet_manager.cc:1398] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:51:33.132804 26980 tablet_copy_service.cc:342] P 297de823852d4a8da44791ab03131cd3: Request end of tablet copy session c107c43e538d48f58625c4320f0aa4b8-a2b61909194d488682afd91c0f796405 received from {username='slave'} at 127.19.66.131:58105
I20240502 20:51:33.132882 26980 tablet_copy_service.cc:433] P 297de823852d4a8da44791ab03131cd3: ending tablet copy session c107c43e538d48f58625c4320f0aa4b8-a2b61909194d488682afd91c0f796405 on tablet a2b61909194d488682afd91c0f796405 with peer c107c43e538d48f58625c4320f0aa4b8
W20240502 20:51:33.148872 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:51:33.153410 18686 ts_tablet_manager.cc:902] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:51:33.153947 18686 tablet_copy_client.cc:308] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:51:33.154511 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet fdd2311ec0b645f5a79098a32c6deba2 from peer c107c43e538d48f58625c4320f0aa4b8 ({username='slave'} at 127.19.66.131:40809)
I20240502 20:51:33.154634 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet fdd2311ec0b645f5a79098a32c6deba2 from peer c107c43e538d48f58625c4320f0aa4b8 at {username='slave'} at 127.19.66.131:40809: session id = c107c43e538d48f58625c4320f0aa4b8-fdd2311ec0b645f5a79098a32c6deba2
I20240502 20:51:33.156167 27112 tablet_copy_source_session.cc:205] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:33.157166 18686 ts_tablet_manager.cc:1875] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:33.159134 18686 ts_tablet_manager.cc:1888] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240502 20:51:33.159312 18686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd2311ec0b645f5a79098a32c6deba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:33.160346 18686 tablet_copy_client.cc:783] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 5 data blocks...
W20240502 20:51:33.165114 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:51:33.166678 18686 tablet_copy_client.cc:647] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:33.168363 18686 tablet_copy_client.cc:523] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:33.170009 18686 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:51:33.180419 18686 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:33.180780 18686 tablet_bootstrap.cc:492] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap complete.
I20240502 20:51:33.180972 18686 ts_tablet_manager.cc:1367] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.003s
I20240502 20:51:33.181489 18686 raft_consensus.cc:348] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:51:33.181622 18686 raft_consensus.cc:729] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: LEARNER
I20240502 20:51:33.181706 18686 consensus_queue.cc:260] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:51:33.182004 18686 ts_tablet_manager.cc:1398] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.003s	sys 0.000s
I20240502 20:51:33.182727 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session c107c43e538d48f58625c4320f0aa4b8-fdd2311ec0b645f5a79098a32c6deba2 received from {username='slave'} at 127.19.66.131:40809
I20240502 20:51:33.182821 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session c107c43e538d48f58625c4320f0aa4b8-fdd2311ec0b645f5a79098a32c6deba2 on tablet fdd2311ec0b645f5a79098a32c6deba2 with peer c107c43e538d48f58625c4320f0aa4b8
I20240502 20:51:33.186326 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(5cf737e20ca74db1b94ccff5d60133ee): 4586 bytes on disk
I20240502 20:51:33.186923 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(5cf737e20ca74db1b94ccff5d60133ee) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 20:51:33.213774 18758 ts_tablet_manager.cc:902] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:51:33.214665 18758 tablet_copy_client.cc:308] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:51:33.215226 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba from peer ebd8300729ec4226acc38b9cff2d8d3b ({username='slave'} at 127.19.66.132:52445)
I20240502 20:51:33.215365 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba from peer ebd8300729ec4226acc38b9cff2d8d3b at {username='slave'} at 127.19.66.132:52445: session id = ebd8300729ec4226acc38b9cff2d8d3b-62c71a33fbcc4cfdbf0065a3fc9dfeba
I20240502 20:51:33.216925 27249 tablet_copy_source_session.cc:205] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 5 blocks and 1 log segments
W20240502 20:51:33.217931 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:33.218282 18758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:33.220803 18758 tablet_copy_client.cc:783] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:33.226684 18758 tablet_copy_client.cc:647] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:33.228147 18758 tablet_copy_client.cc:523] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:33.230078 18758 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:51:33.240074 18758 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:33.240484 18758 tablet_bootstrap.cc:492] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap complete.
I20240502 20:51:33.240749 18758 ts_tablet_manager.cc:1367] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.000s
I20240502 20:51:33.241365 18758 raft_consensus.cc:348] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:51:33.241519 18758 raft_consensus.cc:729] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: LEARNER
I20240502 20:51:33.241640 18758 consensus_queue.cc:260] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
W20240502 20:51:33.242005 27160 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 20:51:33.241998 18758 ts_tablet_manager.cc:1398] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240502 20:51:33.243194 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session ebd8300729ec4226acc38b9cff2d8d3b-62c71a33fbcc4cfdbf0065a3fc9dfeba received from {username='slave'} at 127.19.66.132:52445
I20240502 20:51:33.243281 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session ebd8300729ec4226acc38b9cff2d8d3b-62c71a33fbcc4cfdbf0065a3fc9dfeba on tablet 62c71a33fbcc4cfdbf0065a3fc9dfeba with peer ebd8300729ec4226acc38b9cff2d8d3b
I20240502 20:51:33.249652 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(fdd2311ec0b645f5a79098a32c6deba2): 4840 bytes on disk
I20240502 20:51:33.251005 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(fdd2311ec0b645f5a79098a32c6deba2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 20:51:33.252435 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(a2b61909194d488682afd91c0f796405): 4586 bytes on disk
I20240502 20:51:33.253288 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(a2b61909194d488682afd91c0f796405) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240502 20:51:33.255491 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:33.268232 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(62c71a33fbcc4cfdbf0065a3fc9dfeba): 4925 bytes on disk
I20240502 20:51:33.268784 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(62c71a33fbcc4cfdbf0065a3fc9dfeba) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 20:51:33.272874 27227 raft_consensus.cc:1203] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 20:51:33.278113 18729 ts_tablet_manager.cc:902] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:51:33.278664 18729 tablet_copy_client.cc:308] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:51:33.279299 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet f1545142b0bd40859bf33bf8686002ae from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:45531)
I20240502 20:51:33.279421 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet f1545142b0bd40859bf33bf8686002ae from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:45531: session id = 297de823852d4a8da44791ab03131cd3-f1545142b0bd40859bf33bf8686002ae
I20240502 20:51:33.280665 27249 tablet_copy_source_session.cc:205] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:33.281710 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1545142b0bd40859bf33bf8686002ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:33.284426 18729 tablet_copy_client.cc:783] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:33.289949 18729 tablet_copy_client.cc:647] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:33.291965 18729 tablet_copy_client.cc:523] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:33.294167 18729 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:51:33.303640 18729 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:33.304050 18729 tablet_bootstrap.cc:492] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:51:33.304291 18729 ts_tablet_manager.cc:1367] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240502 20:51:33.304917 18729 raft_consensus.cc:348] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:33.305132 18729 raft_consensus.cc:729] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:51:33.305276 18729 consensus_queue.cc:260] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:33.305648 18729 ts_tablet_manager.cc:1398] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240502 20:51:33.306501 27249 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-f1545142b0bd40859bf33bf8686002ae received from {username='slave'} at 127.19.66.129:45531
I20240502 20:51:33.306622 27249 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 297de823852d4a8da44791ab03131cd3-f1545142b0bd40859bf33bf8686002ae on tablet f1545142b0bd40859bf33bf8686002ae with peer 297de823852d4a8da44791ab03131cd3
I20240502 20:51:33.374526 26960 raft_consensus.cc:1203] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20240502 20:51:33.376816 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:33.378495 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:51:33.400655 17470 consensus_queue.cc:579] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.209s)
W20240502 20:51:33.402379 27025 consensus_peers.cc:462] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:51:33.404146 27090 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } }
I20240502 20:51:33.405340 18729 ts_tablet_manager.cc:902] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:51:33.405925 18729 tablet_copy_client.cc:308] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:51:33.406013 26960 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:51:33.406461 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 843a26fc48844bfd82911b6a27833023 from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:46731)
I20240502 20:51:33.406586 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 843a26fc48844bfd82911b6a27833023 from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:46731: session id = 297de823852d4a8da44791ab03131cd3-843a26fc48844bfd82911b6a27833023
I20240502 20:51:33.406731 18814 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:51:33.407799 27382 tablet_copy_source_session.cc:205] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 5 blocks and 1 log segments
W20240502 20:51:33.408342 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.408746 18729 ts_tablet_manager.cc:1875] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:33.409188 18622 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:51:33.409745 26960 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } }
I20240502 20:51:33.410935 18729 ts_tablet_manager.cc:1888] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:51:33.411299 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843a26fc48844bfd82911b6a27833023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:33.413159 26805 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 9 to 10, NON_VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:33.413544 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f00c4b7954564820875a25da5834416a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:33.413033 18729 tablet_copy_client.cc:783] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:33.417239 27090 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
W20240502 20:51:33.419428 27026 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): Couldn't send request to peer ebd8300729ec4226acc38b9cff2d8d3b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.419915 26959 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 20:51:33.420248 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.420508 18622 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:51:33.421213 18729 tablet_copy_client.cc:647] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:33.422467 17470 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:51:33.422921 26960 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:51:33.424261 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f00c4b7954564820875a25da5834416a with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:33.425267 18729 tablet_copy_client.cc:523] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:33.425479 26805 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 10 to 11, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:51:33.426865 18729 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
W20240502 20:51:33.431347 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:51:33.434880 18729 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:33.435202 18729 tablet_bootstrap.cc:492] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:51:33.435421 18729 ts_tablet_manager.cc:1367] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240502 20:51:33.435923 18729 raft_consensus.cc:348] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:33.436038 18729 raft_consensus.cc:729] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:51:33.436120 18729 consensus_queue.cc:260] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:33.436347 18729 ts_tablet_manager.cc:1398] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:51:33.436933 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-843a26fc48844bfd82911b6a27833023 received from {username='slave'} at 127.19.66.129:46731
I20240502 20:51:33.437028 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 297de823852d4a8da44791ab03131cd3-843a26fc48844bfd82911b6a27833023 on tablet 843a26fc48844bfd82911b6a27833023 with peer 297de823852d4a8da44791ab03131cd3
I20240502 20:51:33.439597 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(f1545142b0bd40859bf33bf8686002ae): 4925 bytes on disk
I20240502 20:51:33.440130 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(f1545142b0bd40859bf33bf8686002ae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 20:51:33.441571 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(843a26fc48844bfd82911b6a27833023): 4586 bytes on disk
I20240502 20:51:33.442051 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(843a26fc48844bfd82911b6a27833023) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240502 20:51:33.442112 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:33.514281 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:51:33.563697 18682 consensus_queue.cc:579] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.017s)
W20240502 20:51:33.565236 26891 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:33.567859 26960 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:51:33.570752 27090 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:33.571931 18879 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 20:51:33.572968 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.575115 18880 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:51:33.575670 27090 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:51:33.577344 26794 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af693f5816e046f592fbd9b2e7ef0ad9 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:33.579259 26806 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 7 to 8, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:33.583878 26960 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:51:33.586480 27090 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 20:51:33.586536 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.586961 18879 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:51:33.587158 26893 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): Couldn't send request to peer ebd8300729ec4226acc38b9cff2d8d3b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af693f5816e046f592fbd9b2e7ef0ad9. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.588897 18880 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:51:33.589251 27090 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:51:33.591151 26794 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet af693f5816e046f592fbd9b2e7ef0ad9 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:33.591998 18681 consensus_queue.cc:579] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.082s)
I20240502 20:51:33.592367 26805 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 8 to 9, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
W20240502 20:51:33.594262 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:51:33.603724 27227 raft_consensus.cc:1203] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
I20240502 20:51:33.625962 18587 consensus_queue.cc:579] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.314s)
W20240502 20:51:33.627449 27297 consensus_peers.cc:462] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:51:33.629009 27361 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:51:33.630640 27090 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:33.631114 18587 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 20:51:33.631811 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.633482 18503 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:51:33.633972 27090 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:51:33.635412 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5586b9e60b544aed87c24f7b3ce0198c with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:33.636837 26802 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 7 to 8, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:51:33.642647 27361 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
W20240502 20:51:33.644825 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.645164 27090 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 20:51:33.645252 27295 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5586b9e60b544aed87c24f7b3ce0198c. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.645668 18587 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:33.648042 18587 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:33.650291 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5586b9e60b544aed87c24f7b3ce0198c with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:33.650349 27090 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:33.650810 17470 consensus_queue.cc:579] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.157s)
I20240502 20:51:33.652686 26806 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 8 to 9, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:33.656006 27090 consensus_queue.cc:237] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:33.658814 26960 raft_consensus.cc:1203] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 20:51:33.661897 27227 raft_consensus.cc:1203] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 20:51:33.662123 27226 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:33.662546 18821 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:51:33.662757 27024 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:33.663286 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.668293 18622 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:33.669730 27226 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:33.670985 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:33.673539 26806 catalog_manager.cc:5484] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 8 to 9, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:33.715297 27361 raft_consensus.cc:1203] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
I20240502 20:51:33.784140 18587 raft_consensus.cc:1050] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:51:33.784664 18587 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:33.786417 27226 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:51:33.786883 26959 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:51:33.786952 17519 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:51:33.787300 18503 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 20:51:33.788216 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.790184 17519 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:33.790735 27226 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:33.792793 26959 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:33.793491 26806 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 11 to 12, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:33.815685 17470 consensus_queue.cc:579] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.276s)
I20240502 20:51:33.840237 26959 raft_consensus.cc:1203] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 20:51:33.840587 18929 consensus_queue.cc:579] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.079s)
I20240502 20:51:33.843408 27226 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:51:33.844931 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240502 20:51:33.845343 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.845600 18681 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:51:33.845883 17470 consensus_queue.cc:579] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.283s)
I20240502 20:51:33.847857 18680 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:51:33.848531 27361 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } }
I20240502 20:51:33.850507 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bf189c19164442e5af0827f1fb7dc085 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:33.851676 26805 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 11 to 12, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:33.856170 27226 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
W20240502 20:51:33.857992 27161 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): Couldn't send request to peer d1f3f8e444364d6faa42d651b5dbb997. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.858000 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:51:33.858481 18681 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 20:51:33.858985 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.860759 18929 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:51:33.861373 27361 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:51:33.862847 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bf189c19164442e5af0827f1fb7dc085 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:33.864411 26805 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 12 to 13, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:33.867048 18814 consensus_queue.cc:579] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.103s)
I20240502 20:51:33.886631 18929 consensus_queue.cc:579] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.114s)
I20240502 20:51:33.890631 27226 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } }
I20240502 20:51:33.892565 27361 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 20:51:33.892706 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.893249 18929 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:33.895540 18681 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:51:33.896263 18923 raft_consensus.cc:1050] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER c107c43e538d48f58625c4320f0aa4b8 to VOTER
I20240502 20:51:33.896231 27361 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } }
I20240502 20:51:33.896883 18923 consensus_queue.cc:237] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:51:33.897672 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a906569ab3b14520b4876bc8d5450f7d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:33.899015 27226 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:33.899035 26806 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 8 to 9, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:33.899261 27090 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:33.899458 18923 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:51:33.899767 18503 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 20:51:33.900816 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.903142 26959 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:33.903883 18923 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:51:33.903883 18587 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:33.905977 26959 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:33.907341 27226 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:51:33.909273 26806 catalog_manager.cc:5484] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 7 to 8, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:33.910238 27361 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 20:51:33.910745 27161 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): Couldn't send request to peer d1f3f8e444364d6faa42d651b5dbb997. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.910637 27091 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:33.911047 18929 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:51:33.911226 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:33.913040 27227 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:33.913213 18681 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:51:33.914839 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a906569ab3b14520b4876bc8d5450f7d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:33.917326 27361 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:51:33.918030 26805 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 9 to 10, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:33.945905 19004 ts_tablet_manager.cc:902] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:51:33.946592 19004 tablet_copy_client.cc:308] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:51:33.947224 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet f00c4b7954564820875a25da5834416a from peer ebd8300729ec4226acc38b9cff2d8d3b ({username='slave'} at 127.19.66.132:50941)
I20240502 20:51:33.947352 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet f00c4b7954564820875a25da5834416a from peer ebd8300729ec4226acc38b9cff2d8d3b at {username='slave'} at 127.19.66.132:50941: session id = ebd8300729ec4226acc38b9cff2d8d3b-f00c4b7954564820875a25da5834416a
I20240502 20:51:33.948910 27112 tablet_copy_source_session.cc:205] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:33.950273 19004 ts_tablet_manager.cc:1875] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:33.953168 19004 ts_tablet_manager.cc:1888] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:51:33.953512 19004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00c4b7954564820875a25da5834416a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 20:51:33.954866 26796 catalog_manager.cc:4573] TS 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bf189c19164442e5af0827f1fb7dc085: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:51:33.955019 19004 tablet_copy_client.cc:783] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Starting download of 5 data blocks...
W20240502 20:51:33.957304 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:33.962353 19004 tablet_copy_client.cc:647] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:33.964279 19004 tablet_copy_client.cc:523] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:33.966291 19004 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:51:33.979368 19004 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:33.979786 19004 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap complete.
I20240502 20:51:33.980073 19004 ts_tablet_manager.cc:1367] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.014s	user 0.010s	sys 0.004s
I20240502 20:51:33.980531 19004 raft_consensus.cc:348] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:51:33.980677 19004 raft_consensus.cc:729] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: LEARNER
I20240502 20:51:33.980796 19004 consensus_queue.cc:260] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:51:33.981760 19004 ts_tablet_manager.cc:1398] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:51:33.983078 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session ebd8300729ec4226acc38b9cff2d8d3b-f00c4b7954564820875a25da5834416a received from {username='slave'} at 127.19.66.132:50941
I20240502 20:51:33.983183 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session ebd8300729ec4226acc38b9cff2d8d3b-f00c4b7954564820875a25da5834416a on tablet f00c4b7954564820875a25da5834416a with peer ebd8300729ec4226acc38b9cff2d8d3b
I20240502 20:51:34.019687 19004 ts_tablet_manager.cc:902] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b: Initiating tablet copy from peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:51:34.020313 19004 tablet_copy_client.cc:308] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:38835
I20240502 20:51:34.021042 26980 tablet_copy_service.cc:140] P 297de823852d4a8da44791ab03131cd3: Received BeginTabletCopySession request for tablet af693f5816e046f592fbd9b2e7ef0ad9 from peer ebd8300729ec4226acc38b9cff2d8d3b ({username='slave'} at 127.19.66.132:51721)
I20240502 20:51:34.021288 26980 tablet_copy_service.cc:161] P 297de823852d4a8da44791ab03131cd3: Beginning new tablet copy session on tablet af693f5816e046f592fbd9b2e7ef0ad9 from peer ebd8300729ec4226acc38b9cff2d8d3b at {username='slave'} at 127.19.66.132:51721: session id = ebd8300729ec4226acc38b9cff2d8d3b-af693f5816e046f592fbd9b2e7ef0ad9
I20240502 20:51:34.022636 26980 tablet_copy_source_session.cc:205] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:34.022678 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(f00c4b7954564820875a25da5834416a): 4925 bytes on disk
I20240502 20:51:34.023283 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(f00c4b7954564820875a25da5834416a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240502 20:51:34.024040 19004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693f5816e046f592fbd9b2e7ef0ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:34.026670 19004 tablet_copy_client.cc:783] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:34.030241 18681 consensus_queue.cc:579] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.362s)
I20240502 20:51:34.033579 19004 tablet_copy_client.cc:647] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:34.033453 27226 consensus_queue.cc:237] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
W20240502 20:51:34.035545 27161 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): Couldn't send request to peer d1f3f8e444364d6faa42d651b5dbb997. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:34.035575 26959 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 20:51:34.036010 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:34.036144 19003 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:51:34.037518 19004 tablet_copy_client.cc:523] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:34.038565 18681 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:51:34.038990 26959 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:51:34.039536 19004 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:51:34.042284 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b466a750cc6b4382947282d8aa6aeb1f with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:34.042480 26805 catalog_manager.cc:5484] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 6 to 7, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:51:34.052609 19004 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:34.052995 19004 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap complete.
I20240502 20:51:34.053287 19004 ts_tablet_manager.cc:1367] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.014s	user 0.014s	sys 0.000s
I20240502 20:51:34.053848 19004 raft_consensus.cc:348] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:51:34.054036 19004 raft_consensus.cc:729] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: LEARNER
I20240502 20:51:34.054185 19004 consensus_queue.cc:260] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
W20240502 20:51:34.054344 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:34.054524 19004 ts_tablet_manager.cc:1398] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:51:34.055140 26980 tablet_copy_service.cc:342] P 297de823852d4a8da44791ab03131cd3: Request end of tablet copy session ebd8300729ec4226acc38b9cff2d8d3b-af693f5816e046f592fbd9b2e7ef0ad9 received from {username='slave'} at 127.19.66.132:51721
I20240502 20:51:34.055261 26980 tablet_copy_service.cc:433] P 297de823852d4a8da44791ab03131cd3: ending tablet copy session ebd8300729ec4226acc38b9cff2d8d3b-af693f5816e046f592fbd9b2e7ef0ad9 on tablet af693f5816e046f592fbd9b2e7ef0ad9 with peer ebd8300729ec4226acc38b9cff2d8d3b
I20240502 20:51:34.062115 18821 consensus_queue.cc:579] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 for more than 300 seconds (300.298s)
I20240502 20:51:34.089845 18894 raft_consensus.cc:1050] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3: attempting to promote NON_VOTER c107c43e538d48f58625c4320f0aa4b8 to VOTER
I20240502 20:51:34.090340 18894 consensus_queue.cc:237] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:51:34.092093 27227 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:34.092404 27361 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:34.092589 18880 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:51:34.092801 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:34.092839 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:34.093029 18682 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:34.095131 18880 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.095705 27361 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.098603 27227 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.099203 26806 catalog_manager.cc:5484] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 8 to 9, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.103480 19053 ts_tablet_manager.cc:902] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:51:34.104270 19053 tablet_copy_client.cc:308] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:51:34.104871 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:37781)
I20240502 20:51:34.104986 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:37781: session id = 297de823852d4a8da44791ab03131cd3-62ebc6576a4e4a108de55a4e9d7ff6b8
I20240502 20:51:34.106155 27112 tablet_copy_source_session.cc:205] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:34.107328 19053 ts_tablet_manager.cc:1875] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:34.110177 19053 ts_tablet_manager.cc:1888] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 20:51:34.110363 19053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:34.111486 19053 tablet_copy_client.cc:783] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:34.116427 19053 tablet_copy_client.cc:647] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:34.118394 19053 tablet_copy_client.cc:523] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:34.120532 19053 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:51:34.125779 19058 ts_tablet_manager.cc:902] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:51:34.127224 19058 tablet_copy_client.cc:308] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:51:34.127854 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 5586b9e60b544aed87c24f7b3ce0198c from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:46731)
I20240502 20:51:34.127986 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 5586b9e60b544aed87c24f7b3ce0198c from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:46731: session id = 297de823852d4a8da44791ab03131cd3-5586b9e60b544aed87c24f7b3ce0198c
I20240502 20:51:34.129240 27382 tablet_copy_source_session.cc:205] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:34.130494 19058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5586b9e60b544aed87c24f7b3ce0198c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:34.130930 17470 raft_consensus.cc:1050] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER c107c43e538d48f58625c4320f0aa4b8 to VOTER
I20240502 20:51:34.131397 17470 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:51:34.132341 19053 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:34.132704 19053 tablet_bootstrap.cc:492] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:51:34.132886 19053 ts_tablet_manager.cc:1367] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.012s	user 0.008s	sys 0.000s
I20240502 20:51:34.133406 19053 raft_consensus.cc:348] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:34.133533 19053 raft_consensus.cc:729] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:51:34.133613 26959 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:51:34.133637 19053 consensus_queue.cc:260] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:34.133771 27226 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:51:34.133947 19053 ts_tablet_manager.cc:1398] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240502 20:51:34.134423 18681 raft_consensus.cc:1050] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b to VOTER
I20240502 20:51:34.134531 18821 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:51:34.134759 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-62ebc6576a4e4a108de55a4e9d7ff6b8 received from {username='slave'} at 127.19.66.129:37781
I20240502 20:51:34.134829 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 297de823852d4a8da44791ab03131cd3-62ebc6576a4e4a108de55a4e9d7ff6b8 on tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 with peer 297de823852d4a8da44791ab03131cd3
I20240502 20:51:34.134882 18821 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:51:34.134932 18681 consensus_queue.cc:237] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } }
W20240502 20:51:34.136173 27024 consensus_peers.cc:485] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:34.136550 19058 tablet_copy_client.cc:783] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:34.137105 27091 raft_consensus.cc:1261] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:51:34.137300 26960 raft_consensus.cc:1261] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:51:34.137660 27361 raft_consensus.cc:1261] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:51:34.138123 19002 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:51:34.139391 19003 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:51:34.139781 19003 consensus_queue.cc:1035] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:51:34.142424 19058 tablet_copy_client.cc:647] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:34.143910 19058 tablet_copy_client.cc:523] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:34.143796 19003 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:34.144311 18622 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.145743 19058 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:51:34.147910 26802 catalog_manager.cc:5484] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 6 to 7, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:34.147948 26959 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:34.148564 26960 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.152040 26805 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 11 to 12, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.153036 18681 raft_consensus.cc:1050] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:51:34.152889 27226 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.153769 18681 consensus_queue.cc:237] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:34.156333 26960 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 20:51:34.157379 27158 consensus_peers.cc:485] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:34.158500 27090 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:34.159037 18929 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:51:34.159761 27361 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:34.162626 18929 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:51:34.165452 27091 raft_consensus.cc:2932] T 62c71a33fbcc4cfdbf0065a3fc9dfeba P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:34.170325 27091 consensus_queue.cc:237] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:51:34.175449 19058 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:34.175870 19058 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:51:34.176110 19058 ts_tablet_manager.cc:1367] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.030s	user 0.008s	sys 0.004s
I20240502 20:51:34.177035 27226 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:51:34.177343 19058 raft_consensus.cc:348] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:34.177485 19058 raft_consensus.cc:729] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:51:34.177639 26959 raft_consensus.cc:1261] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:51:34.177574 17470 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:51:34.177595 19058 consensus_queue.cc:260] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:34.178063 19058 ts_tablet_manager.cc:1398] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 20:51:34.178813 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-5586b9e60b544aed87c24f7b3ce0198c received from {username='slave'} at 127.19.66.129:46731
I20240502 20:51:34.178828 17470 consensus_queue.cc:1035] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:51:34.178903 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 297de823852d4a8da44791ab03131cd3-5586b9e60b544aed87c24f7b3ce0198c on tablet 5586b9e60b544aed87c24f7b3ce0198c with peer 297de823852d4a8da44791ab03131cd3
I20240502 20:51:34.180006 18929 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.181362 17470 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.183753 26960 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.184643 27226 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.187065 27091 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.187283 26802 catalog_manager.cc:5484] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 7 to 8, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:34.187620 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fdd2311ec0b645f5a79098a32c6deba2 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:34.187635 26959 raft_consensus.cc:2932] T fdd2311ec0b645f5a79098a32c6deba2 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.191989 27226 consensus_queue.cc:237] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:34.193040 26802 catalog_manager.cc:5484] T fdd2311ec0b645f5a79098a32c6deba2 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 12 to 13, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.194186 26959 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:34.194727 18681 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:34.196628 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(62ebc6576a4e4a108de55a4e9d7ff6b8): 4923 bytes on disk
I20240502 20:51:34.197266 27090 raft_consensus.cc:1261] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:34.197228 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(62ebc6576a4e4a108de55a4e9d7ff6b8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 20:51:34.197971 18681 consensus_queue.cc:1035] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:34.198338 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(5586b9e60b544aed87c24f7b3ce0198c): 4840 bytes on disk
I20240502 20:51:34.199950 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(5586b9e60b544aed87c24f7b3ce0198c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 20:51:34.200186 19003 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.200721 27091 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.202512 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1545142b0bd40859bf33bf8686002ae with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:34.204218 26960 raft_consensus.cc:2932] T f1545142b0bd40859bf33bf8686002ae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.204917 26805 catalog_manager.cc:5484] T f1545142b0bd40859bf33bf8686002ae P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 8 to 9, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:34.271302 18923 raft_consensus.cc:1050] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:51:34.272010 18923 consensus_queue.cc:237] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:34.274338 27226 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:34.274338 26960 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:34.274828 17519 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:51:34.275231 17519 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:51:34.275508 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(af693f5816e046f592fbd9b2e7ef0ad9): 4840 bytes on disk
I20240502 20:51:34.276111 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(af693f5816e046f592fbd9b2e7ef0ad9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240502 20:51:34.277350 27295 consensus_peers.cc:485] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:34.279806 18503 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.280539 26960 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.282315 27226 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.282953 26805 catalog_manager.cc:5484] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 7 to 8, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:34.288326 27361 consensus_queue.cc:237] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:34.290608 26960 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:34.291123 27226 raft_consensus.cc:1261] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:34.291108 18923 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:34.291719 18923 consensus_queue.cc:1035] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:34.293779 18587 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.294567 26960 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.294802 27226 raft_consensus.cc:2932] T 843a26fc48844bfd82911b6a27833023 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:34.295786 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 843a26fc48844bfd82911b6a27833023 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:34.297122 26806 catalog_manager.cc:5484] T 843a26fc48844bfd82911b6a27833023 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 8 to 9, VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:51:34.338652 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:34.357296 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:51:34.375612 19109 ts_tablet_manager.cc:902] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:51:34.376345 19109 tablet_copy_client.cc:308] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:51:34.377254 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet a906569ab3b14520b4876bc8d5450f7d from peer d1f3f8e444364d6faa42d651b5dbb997 ({username='slave'} at 127.19.66.130:60245)
I20240502 20:51:34.377403 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet a906569ab3b14520b4876bc8d5450f7d from peer d1f3f8e444364d6faa42d651b5dbb997 at {username='slave'} at 127.19.66.130:60245: session id = d1f3f8e444364d6faa42d651b5dbb997-a906569ab3b14520b4876bc8d5450f7d
I20240502 20:51:34.378420 27249 tablet_copy_source_session.cc:205] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:34.379665 19109 ts_tablet_manager.cc:1875] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:34.382205 19109 ts_tablet_manager.cc:1888] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:51:34.382422 19109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a906569ab3b14520b4876bc8d5450f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:34.383684 19109 tablet_copy_client.cc:783] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:34.387971 19114 ts_tablet_manager.cc:902] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:51:34.388833 19114 tablet_copy_client.cc:308] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:51:34.389611 27249 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet bf189c19164442e5af0827f1fb7dc085 from peer d1f3f8e444364d6faa42d651b5dbb997 ({username='slave'} at 127.19.66.130:60245)
I20240502 20:51:34.389765 27249 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet bf189c19164442e5af0827f1fb7dc085 from peer d1f3f8e444364d6faa42d651b5dbb997 at {username='slave'} at 127.19.66.130:60245: session id = d1f3f8e444364d6faa42d651b5dbb997-bf189c19164442e5af0827f1fb7dc085
I20240502 20:51:34.390770 17470 consensus_queue.cc:579] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.496s)
I20240502 20:51:34.391623 19109 tablet_copy_client.cc:647] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:34.392931 27249 tablet_copy_source_session.cc:205] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:34.395375 19109 tablet_copy_client.cc:523] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:34.397454 19109 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:51:34.397671 19114 ts_tablet_manager.cc:1875] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:34.400220 19114 ts_tablet_manager.cc:1888] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 20:51:34.400419 19114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf189c19164442e5af0827f1fb7dc085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:34.401721 19114 tablet_copy_client.cc:783] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:34.410307 19109 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:34.410741 19109 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap complete.
I20240502 20:51:34.411011 19109 ts_tablet_manager.cc:1367] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.014s	user 0.011s	sys 0.000s
I20240502 20:51:34.411554 19109 raft_consensus.cc:348] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:51:34.411698 19109 raft_consensus.cc:729] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: LEARNER
I20240502 20:51:34.411849 19114 tablet_copy_client.cc:647] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:34.411885 19109 consensus_queue.cc:260] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:51:34.412333 19109 ts_tablet_manager.cc:1398] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.003s	sys 0.000s
I20240502 20:51:34.412909 27248 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session d1f3f8e444364d6faa42d651b5dbb997-a906569ab3b14520b4876bc8d5450f7d received from {username='slave'} at 127.19.66.130:60245
I20240502 20:51:34.413028 27248 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session d1f3f8e444364d6faa42d651b5dbb997-a906569ab3b14520b4876bc8d5450f7d on tablet a906569ab3b14520b4876bc8d5450f7d with peer d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:51:34.415316 19114 tablet_copy_client.cc:523] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:34.417922 19114 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:51:34.418287 18929 consensus_queue.cc:579] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.335s)
I20240502 20:51:34.418957 27361 consensus_queue.cc:237] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:51:34.421316 26960 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:34.421952 18587 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 20:51:34.422473 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:34.422801 27226 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:34.423740 18587 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:34.424455 18923 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.424994 26960 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.426822 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40df15a82a624913b3b712687d999134 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:34.428447 26802 catalog_manager.cc:5484] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 8 to 9, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.429111 27226 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:51:34.432365 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet 40df15a82a624913b3b712687d999134 with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47758
W20240502 20:51:34.432677 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:51:34.432936 19121 tablet_replica.cc:330] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:51:34.433117 19121 raft_consensus.cc:2229] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:51:34.433449 19121 raft_consensus.cc:2258] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:51:34.434659 19114 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:34.435165 19114 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap complete.
I20240502 20:51:34.435489 19114 ts_tablet_manager.cc:1367] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.010s
I20240502 20:51:34.435894 19121 ts_tablet_manager.cc:1875] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:51:34.436064 19114 raft_consensus.cc:348] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:51:34.436234 19114 raft_consensus.cc:729] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: LEARNER
I20240502 20:51:34.436367 19114 consensus_queue.cc:260] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:51:34.436697 19114 ts_tablet_manager.cc:1398] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:51:34.437528 27248 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session d1f3f8e444364d6faa42d651b5dbb997-bf189c19164442e5af0827f1fb7dc085 received from {username='slave'} at 127.19.66.130:60245
I20240502 20:51:34.437654 27248 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session d1f3f8e444364d6faa42d651b5dbb997-bf189c19164442e5af0827f1fb7dc085 on tablet bf189c19164442e5af0827f1fb7dc085 with peer d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:51:34.440685 19121 ts_tablet_manager.cc:1888] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 20:51:34.440785 19121 log.cc:1196] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/40df15a82a624913b3b712687d999134
I20240502 20:51:34.441606 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet 40df15a82a624913b3b712687d999134 (table rebalance_test_table_8 [id=2bfbb94d5cdf44cb82f42848d797bd88]) successfully deleted
I20240502 20:51:34.451771 19002 consensus_queue.cc:579] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.368s)
I20240502 20:51:34.455207 27226 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:51:34.456816 27091 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:34.457371 18929 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:51:34.458492 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:34.459420 18681 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:34.460376 27091 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:34.462054 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84484d65e2bb4e23ac621a39434ce382 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:34.462848 26802 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 10 to 11, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:34.467450 27226 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
W20240502 20:51:34.470407 27158 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:34.470889 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:34.471092 27091 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:51:34.471493 18681 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:51:34.473932 18929 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
W20240502 20:51:34.474642 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:34.476141 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 84484d65e2bb4e23ac621a39434ce382 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:34.476541 27091 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:51:34.478085 26806 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 11 to 12, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:34.486564 27361 raft_consensus.cc:1203] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
I20240502 20:51:34.524747 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(bf189c19164442e5af0827f1fb7dc085): 4925 bytes on disk
I20240502 20:51:34.525314 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(bf189c19164442e5af0827f1fb7dc085) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240502 20:51:34.526399 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(a906569ab3b14520b4876bc8d5450f7d): 4586 bytes on disk
I20240502 20:51:34.526854 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(a906569ab3b14520b4876bc8d5450f7d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240502 20:51:34.545729 27025 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:34.630012 19114 ts_tablet_manager.cc:902] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:51:34.630633 19114 tablet_copy_client.cc:308] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:51:34.631098 27248 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet b466a750cc6b4382947282d8aa6aeb1f from peer d1f3f8e444364d6faa42d651b5dbb997 ({username='slave'} at 127.19.66.130:60245)
I20240502 20:51:34.631213 27248 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet b466a750cc6b4382947282d8aa6aeb1f from peer d1f3f8e444364d6faa42d651b5dbb997 at {username='slave'} at 127.19.66.130:60245: session id = d1f3f8e444364d6faa42d651b5dbb997-b466a750cc6b4382947282d8aa6aeb1f
I20240502 20:51:34.631238 27361 raft_consensus.cc:1203] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 20:51:34.632479 27248 tablet_copy_source_session.cc:205] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:34.633481 19114 ts_tablet_manager.cc:1875] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:34.635989 19114 ts_tablet_manager.cc:1888] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:51:34.636157 19114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b466a750cc6b4382947282d8aa6aeb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:34.637147 19114 tablet_copy_client.cc:783] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:34.643947 19114 tablet_copy_client.cc:647] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:34.645691 19114 tablet_copy_client.cc:523] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:34.647835 19114 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:51:34.653991 26960 raft_consensus.cc:1203] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 20:51:34.658946 19114 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:34.659252 19114 tablet_bootstrap.cc:492] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap complete.
I20240502 20:51:34.659464 19114 ts_tablet_manager.cc:1367] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.003s
I20240502 20:51:34.659915 19114 raft_consensus.cc:348] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:51:34.660033 19114 raft_consensus.cc:729] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: LEARNER
I20240502 20:51:34.660120 19114 consensus_queue.cc:260] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:51:34.660593 19114 ts_tablet_manager.cc:1398] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:51:34.661505 27248 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session d1f3f8e444364d6faa42d651b5dbb997-b466a750cc6b4382947282d8aa6aeb1f received from {username='slave'} at 127.19.66.130:60245
I20240502 20:51:34.661623 27248 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session d1f3f8e444364d6faa42d651b5dbb997-b466a750cc6b4382947282d8aa6aeb1f on tablet b466a750cc6b4382947282d8aa6aeb1f with peer d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:51:34.710654 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240502 20:51:34.711501 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:34.715188 26959 raft_consensus.cc:1203] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 20:51:34.778820 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(b466a750cc6b4382947282d8aa6aeb1f): 4840 bytes on disk
I20240502 20:51:34.779357 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(b466a750cc6b4382947282d8aa6aeb1f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240502 20:51:34.790709 27091 raft_consensus.cc:1203] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
W20240502 20:51:34.808142 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:34.823824 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 20:51:34.860080 27091 raft_consensus.cc:1203] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 20:51:34.900072 17470 raft_consensus.cc:1050] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b to VOTER
I20240502 20:51:34.900595 17470 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } }
I20240502 20:51:34.902566 27361 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:51:34.903043 26959 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:51:34.903190 19219 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 20:51:34.903465 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:34.903496 18814 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:51:34.906824 18814 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:34.907409 26960 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:34.911738 27361 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:34.912178 26806 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 11 to 12, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
W20240502 20:51:34.949965 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 143 similar messages]
I20240502 20:51:35.000512 18814 consensus_queue.cc:579] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.195s)
I20240502 20:51:35.003782 27091 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:35.005908 26960 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 20:51:35.007108 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:35.007436 18814 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:51:35.007810 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:35.008370 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 843a26fc48844bfd82911b6a27833023: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:35.008703 27361 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:35.009271 18814 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:51:35.010054 19219 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.010598 26960 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.013746 26802 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:35.014907 27361 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.017493 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1b9ff3c921d474fa5ad106d1c17ab00 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:35.026551 19245 ts_tablet_manager.cc:902] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:51:35.027418 19245 tablet_copy_client.cc:308] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:51:35.028136 27248 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet 84484d65e2bb4e23ac621a39434ce382 from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:45531)
I20240502 20:51:35.028307 27248 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet 84484d65e2bb4e23ac621a39434ce382 from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:45531: session id = 297de823852d4a8da44791ab03131cd3-84484d65e2bb4e23ac621a39434ce382
I20240502 20:51:35.030129 27248 tablet_copy_source_session.cc:205] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:35.032871 19245 ts_tablet_manager.cc:1875] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:35.036355 19245 ts_tablet_manager.cc:1888] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240502 20:51:35.036937 19245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84484d65e2bb4e23ac621a39434ce382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:35.038141 19245 tablet_copy_client.cc:783] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:35.046309 19245 tablet_copy_client.cc:647] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:35.048445 19245 tablet_copy_client.cc:523] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:35.050313 19245 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:51:35.061609 19245 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:35.061981 19245 tablet_bootstrap.cc:492] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:51:35.062213 19245 ts_tablet_manager.cc:1367] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.012s	user 0.012s	sys 0.000s
I20240502 20:51:35.062716 19245 raft_consensus.cc:348] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:35.062842 19245 raft_consensus.cc:729] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:51:35.063174 19245 consensus_queue.cc:260] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:51:35.063740 19245 ts_tablet_manager.cc:1398] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:51:35.064656 27248 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-84484d65e2bb4e23ac621a39434ce382 received from {username='slave'} at 127.19.66.129:45531
I20240502 20:51:35.064765 27248 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 297de823852d4a8da44791ab03131cd3-84484d65e2bb4e23ac621a39434ce382 on tablet 84484d65e2bb4e23ac621a39434ce382 with peer 297de823852d4a8da44791ab03131cd3
I20240502 20:51:35.111867 18923 raft_consensus.cc:1050] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:51:35.112466 18923 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:35.114570 26957 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:51:35.114799 27091 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 20:51:35.115222 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:35.115671 18923 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:51:35.115867 18587 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:51:35.118815 19000 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.119311 27091 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.122160 26802 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.122435 26957 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.151193 19002 consensus_queue.cc:579] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.319s)
I20240502 20:51:35.155025 27091 raft_consensus.cc:1203] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 20:51:35.176884 18929 consensus_queue.cc:579] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.345s)
W20240502 20:51:35.178989 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 20:51:35.192768 19219 raft_consensus.cc:1050] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:51:35.193362 19219 consensus_queue.cc:237] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:35.195042 26957 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:51:35.195197 27226 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:51:35.196027 17470 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:51:35.196328 18814 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:51:35.196913 27025 consensus_peers.cc:485] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:35.200181 17470 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.200836 27226 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.204398 26957 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.205636 26805 catalog_manager.cc:5484] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 9 to 10, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.206759 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(84484d65e2bb4e23ac621a39434ce382): 4925 bytes on disk
I20240502 20:51:35.207448 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(84484d65e2bb4e23ac621a39434ce382) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 20:51:35.211272 27091 consensus_queue.cc:237] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:35.213497 26957 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:35.213497 27226 raft_consensus.cc:1261] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:35.214182 18814 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:51:35.215085 18814 consensus_queue.cc:1035] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:51:35.216902 19219 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.217654 26957 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.218807 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:35.220335 27226 raft_consensus.cc:2932] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:35.220525 26799 maintenance_manager.cc:419] P e0c668105a124f0eb7227390002cc15e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10357 bytes on disk
I20240502 20:51:35.220547 26805 catalog_manager.cc:5484] T 62ebc6576a4e4a108de55a4e9d7ff6b8 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:35.221223 19190 consensus_queue.cc:579] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.252s)
I20240502 20:51:35.223778 26798 maintenance_manager.cc:643] P e0c668105a124f0eb7227390002cc15e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s	user 0.003s	sys 0.000s Metrics: {"thread_start_us":100,"threads_started":1}
W20240502 20:51:35.224864 26892 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:35.231431 26957 consensus_queue.cc:237] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } }
I20240502 20:51:35.233044 27361 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:35.233330 27226 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:35.233666 19190 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:35.233882 19266 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:35.236644 19266 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:35.237375 27361 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:35.237510 27226 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:35.238924 26794 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2dfc0bbbbca4cc28a97480007343424 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:35.240403 26802 catalog_manager.cc:5484] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 8 to 9, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:35.245414 26957 consensus_queue.cc:237] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:51:35.246187 17470 consensus_queue.cc:579] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.043s)
I20240502 20:51:35.246572 19190 raft_consensus.cc:1050] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: attempting to promote NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b to VOTER
I20240502 20:51:35.247027 19190 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } }
I20240502 20:51:35.248858 27091 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:51:35.249312 27361 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEARNER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:51:35.249686 19266 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:51:35.249812 27091 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
W20240502 20:51:35.250002 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:35.252663 27026 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): Couldn't send request to peer ebd8300729ec4226acc38b9cff2d8d3b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:35.253377 27226 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 3 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 20:51:35.254231 18814 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:51:35.254626 27227 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 20:51:35.255827 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:35.255805 19265 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:51:35.256028 19266 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:51:35.256654 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:35.256795 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:35.257011 26894 consensus_peers.cc:485] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 -> Peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): Couldn't send request to peer d1f3f8e444364d6faa42d651b5dbb997. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:35.257462 18814 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 10 to 13, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:51:35.257855 27361 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:51:35.259603 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a547561343304137a044ce11517f6007 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:35.259586 19266 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:51:35.260326 19266 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:35.260800 19265 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:51:35.261343 26805 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 10 to 13, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New cstate: current_term: 3 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:35.263159 27090 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:35.263183 27361 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:35.263417 26794 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f2dfc0bbbbca4cc28a97480007343424 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:51:35.263677 27227 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:51:35.268297 26807 catalog_manager.cc:5484] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 9 to 10, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:51:35.269404 26806 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 9 to 10, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:35.270776 27227 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 10 to 13, NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } } }
I20240502 20:51:35.277405 27359 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
W20240502 20:51:35.327235 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:35.352382 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 111 similar messages]
I20240502 20:51:35.356359 18587 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:51:35.356554 18587 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:35.356777 18681 raft_consensus.cc:1050] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 to VOTER
I20240502 20:51:35.357260 18587 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:51:35.357319 18681 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:51:35.357511 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:35.358127 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:51:35.359131 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:35.359210 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:51:35.359390 18587 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:51:35.359618 27359 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240502 20:51:35.359731 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:51:35.360143 19307 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:51:35.360213 27090 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240502 20:51:35.360519 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:35.360572 19307 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:51:35.362833 18929 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.363102 27090 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.366395 26805 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 13 to 14, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.366427 27359 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.485566 19307 raft_consensus.cc:1050] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 to VOTER
I20240502 20:51:35.486065 19307 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:51:35.487651 27359 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:35.487836 27090 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:35.488286 19002 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:51:35.488310 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:35.488442 19307 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:51:35.490599 19307 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.491539 27090 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.493758 27359 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.494464 26807 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:51:35.509141 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:35.553490 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:35.600793 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 20:51:35.631546 26957 raft_consensus.cc:1203] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
I20240502 20:51:35.702248 19340 ts_tablet_manager.cc:902] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:51:35.702883 19340 tablet_copy_client.cc:308] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:51:35.703814 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet a547561343304137a044ce11517f6007 from peer ebd8300729ec4226acc38b9cff2d8d3b ({username='slave'} at 127.19.66.132:50941)
I20240502 20:51:35.704046 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet a547561343304137a044ce11517f6007 from peer ebd8300729ec4226acc38b9cff2d8d3b at {username='slave'} at 127.19.66.132:50941: session id = ebd8300729ec4226acc38b9cff2d8d3b-a547561343304137a044ce11517f6007
I20240502 20:51:35.706032 27112 tablet_copy_source_session.cc:205] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:35.707278 19340 ts_tablet_manager.cc:1875] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:35.713701 19340 ts_tablet_manager.cc:1888] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 20:51:35.713966 19340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a547561343304137a044ce11517f6007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:35.715457 19340 tablet_copy_client.cc:783] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:35.721719 19340 tablet_copy_client.cc:647] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:35.723511 19340 tablet_copy_client.cc:523] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:35.725476 19340 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap starting.
I20240502 20:51:35.737473 19340 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:35.737794 19340 tablet_bootstrap.cc:492] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Bootstrap complete.
I20240502 20:51:35.737978 19340 ts_tablet_manager.cc:1367] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent bootstrapping tablet: real 0.013s	user 0.012s	sys 0.000s
I20240502 20:51:35.738430 19340 raft_consensus.cc:348] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:51:35.738572 19340 raft_consensus.cc:729] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: ebd8300729ec4226acc38b9cff2d8d3b, State: Initialized, Role: LEARNER
I20240502 20:51:35.738667 19340 consensus_queue.cc:260] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: NON_VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: true } }
I20240502 20:51:35.738972 19340 ts_tablet_manager.cc:1398] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:51:35.739676 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session ebd8300729ec4226acc38b9cff2d8d3b-a547561343304137a044ce11517f6007 received from {username='slave'} at 127.19.66.132:50941
I20240502 20:51:35.739774 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session ebd8300729ec4226acc38b9cff2d8d3b-a547561343304137a044ce11517f6007 on tablet a547561343304137a044ce11517f6007 with peer ebd8300729ec4226acc38b9cff2d8d3b
I20240502 20:51:35.765998 19307 raft_consensus.cc:1050] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 to VOTER
I20240502 20:51:35.766683 19307 consensus_queue.cc:237] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:51:35.769210 27090 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:35.769357 26957 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:51:35.769971 18929 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:51:35.770260 19002 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 20:51:35.771248 27160 consensus_peers.cc:485] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:35.774158 19002 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.774689 27090 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.777689 26805 catalog_manager.cc:5484] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 7 to 8, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:35.777916 26957 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
W20240502 20:51:35.781818 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:35.783519 27227 consensus_queue.cc:237] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:51:35.785156 26957 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:35.785677 27410 maintenance_manager.cc:419] P ebd8300729ec4226acc38b9cff2d8d3b: Scheduling UndoDeltaBlockGCOp(a547561343304137a044ce11517f6007): 4840 bytes on disk
I20240502 20:51:35.785724 19002 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:35.786068 27090 raft_consensus.cc:1261] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:51:35.786715 19002 consensus_queue.cc:1035] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:51:35.786728 27299 maintenance_manager.cc:643] P ebd8300729ec4226acc38b9cff2d8d3b: UndoDeltaBlockGCOp(a547561343304137a044ce11517f6007) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 20:51:35.788805 19002 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.791373 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b466a750cc6b4382947282d8aa6aeb1f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:35.791363 26957 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.792955 27090 raft_consensus.cc:2932] T b466a750cc6b4382947282d8aa6aeb1f P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:35.793648 26807 catalog_manager.cc:5484] T b466a750cc6b4382947282d8aa6aeb1f P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 8 to 9, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:35.817296 19375 ts_tablet_manager.cc:902] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Initiating tablet copy from peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:51:35.818010 19375 tablet_copy_client.cc:308] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:38835
I20240502 20:51:35.819042 26980 tablet_copy_service.cc:140] P 297de823852d4a8da44791ab03131cd3: Received BeginTabletCopySession request for tablet f2dfc0bbbbca4cc28a97480007343424 from peer d1f3f8e444364d6faa42d651b5dbb997 ({username='slave'} at 127.19.66.130:54199)
I20240502 20:51:35.819205 26980 tablet_copy_service.cc:161] P 297de823852d4a8da44791ab03131cd3: Beginning new tablet copy session on tablet f2dfc0bbbbca4cc28a97480007343424 from peer d1f3f8e444364d6faa42d651b5dbb997 at {username='slave'} at 127.19.66.130:54199: session id = d1f3f8e444364d6faa42d651b5dbb997-f2dfc0bbbbca4cc28a97480007343424
I20240502 20:51:35.820492 26980 tablet_copy_source_session.cc:205] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:51:35.821738 19375 ts_tablet_manager.cc:1875] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:51:35.824331 19375 ts_tablet_manager.cc:1888] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:51:35.824558 19375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2dfc0bbbbca4cc28a97480007343424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:51:35.826547 19375 tablet_copy_client.cc:783] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 5 data blocks...
I20240502 20:51:35.835062 19375 tablet_copy_client.cc:647] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 1 WAL segments...
I20240502 20:51:35.837580 19375 tablet_copy_client.cc:523] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:51:35.839672 19375 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
W20240502 20:51:35.854228 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 20:51:35.854550 19375 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:51:35.855047 19375 tablet_bootstrap.cc:492] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap complete.
I20240502 20:51:35.855335 19375 ts_tablet_manager.cc:1367] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.016s	user 0.012s	sys 0.004s
I20240502 20:51:35.856112 19375 raft_consensus.cc:348] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:51:35.856315 19375 raft_consensus.cc:729] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: LEARNER
I20240502 20:51:35.856540 19375 consensus_queue.cc:260] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:51:35.857131 19375 ts_tablet_manager.cc:1398] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 20:51:35.860657 26980 tablet_copy_service.cc:342] P 297de823852d4a8da44791ab03131cd3: Request end of tablet copy session d1f3f8e444364d6faa42d651b5dbb997-f2dfc0bbbbca4cc28a97480007343424 received from {username='slave'} at 127.19.66.130:54199
I20240502 20:51:35.860818 26980 tablet_copy_service.cc:433] P 297de823852d4a8da44791ab03131cd3: ending tablet copy session d1f3f8e444364d6faa42d651b5dbb997-f2dfc0bbbbca4cc28a97480007343424 on tablet f2dfc0bbbbca4cc28a97480007343424 with peer d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:51:35.884997 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 20:51:35.910319 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:35.967919 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:51:35.988466 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:51:36.029445 19002 raft_consensus.cc:1050] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:51:36.030405 19002 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:36.032263 27090 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:51:36.032263 26957 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:51:36.032972 19307 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:51:36.033362 19002 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 20:51:36.033795 27160 consensus_peers.cc:485] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:36.036345 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(f2dfc0bbbbca4cc28a97480007343424): 4925 bytes on disk
I20240502 20:51:36.036079 19307 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:36.036927 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(f2dfc0bbbbca4cc28a97480007343424) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 20:51:36.037009 26957 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:36.040961 26806 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 12 to 13, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:36.042829 27090 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:36.047638 27227 consensus_queue.cc:237] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:36.049264 27090 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:51:36.049401 26957 raft_consensus.cc:1261] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:51:36.049849 19002 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:51:36.050182 18929 consensus_queue.cc:1035] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:51:36.051697 18681 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:36.052157 27090 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:36.052659 26957 raft_consensus.cc:2932] T 84484d65e2bb4e23ac621a39434ce382 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:51:36.054085 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84484d65e2bb4e23ac621a39434ce382 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:51:36.055076 26807 catalog_manager.cc:5484] T 84484d65e2bb4e23ac621a39434ce382 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 13 to 14, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:51:36.059307 26796 catalog_manager.cc:4573] TS 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 84484d65e2bb4e23ac621a39434ce382: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:36.163106 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:51:36.169922 27359 raft_consensus.cc:1203] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 3 LEARNER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13]   Dedup: 3.13->[]
I20240502 20:51:36.217336 27090 raft_consensus.cc:1203] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
W20240502 20:51:36.613615 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:51:36.635797 19266 raft_consensus.cc:1050] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3: attempting to promote NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 to VOTER
I20240502 20:51:36.636418 19266 consensus_queue.cc:237] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:51:36.638213 27090 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:36.638214 27359 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:36.638470 27227 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:51:36.638758 19466 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:51:36.639299 19411 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:51:36.639823 19466 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:51:36.642827 19412 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:36.643951 27226 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:36.645217 27359 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:36.646291 27090 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:36.646787 26805 catalog_manager.cc:5484] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:51:36.654074 26957 consensus_queue.cc:237] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:51:36.655629 27359 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:51:36.656111 27090 raft_consensus.cc:1261] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:51:36.656157 19466 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:51:36.656577 19266 consensus_queue.cc:1035] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:51:36.658226 19411 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:36.659029 27359 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:36.660954 26794 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2dfc0bbbbca4cc28a97480007343424 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 20:51:36.662705 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:51:36.662864 26807 catalog_manager.cc:5484] T f2dfc0bbbbca4cc28a97480007343424 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 11 to 12, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:51:36.663393 27090 raft_consensus.cc:2932] T f2dfc0bbbbca4cc28a97480007343424 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:51:36.666467 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet f2dfc0bbbbca4cc28a97480007343424 with delete_type TABLET_DATA_TOMBSTONED (TS c107c43e538d48f58625c4320f0aa4b8 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:40306
I20240502 20:51:36.667052 19473 tablet_replica.cc:330] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8: stopping tablet replica
I20240502 20:51:36.667279 19473 raft_consensus.cc:2229] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:51:36.667912 19473 raft_consensus.cc:2258] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:51:36.668984 19473 ts_tablet_manager.cc:1875] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:51:36.672602 19473 ts_tablet_manager.cc:1888] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 20:51:36.672703 19473 log.cc:1196] T f2dfc0bbbbca4cc28a97480007343424 P c107c43e538d48f58625c4320f0aa4b8: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/f2dfc0bbbbca4cc28a97480007343424
I20240502 20:51:36.673491 26797 catalog_manager.cc:4830] TS c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): tablet f2dfc0bbbbca4cc28a97480007343424 (table rebalance_test_table_0 [id=d9dbe3e88393443a8397e83015b69044]) successfully deleted
I20240502 20:51:36.723115 19279 raft_consensus.cc:1050] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER ebd8300729ec4226acc38b9cff2d8d3b to VOTER
I20240502 20:51:36.724073 19279 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } }
I20240502 20:51:36.726864 27359 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 3 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 20:51:36.727800 27226 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 3 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
W20240502 20:51:36.728278 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:51:36.728463 19410 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:51:36.728683 19479 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240502 20:51:36.728683 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:36.730005 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:51:36.732779 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:51:36.733702 19479 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:36.734417 27359 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:36.737200 27226 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:51:36.738340 26807 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 13 to 14, ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: NON_VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:51:36.808276 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 20:51:36.912319 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:36.917120 26796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 156 similar messages]
W20240502 20:51:36.986353 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:37.139258 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:37.200805 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d1b9ff3c921d474fa5ad106d1c17ab00: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:37.286509 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:37.356467 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:37.401175 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 20:51:37.518834 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:37.549950 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:37.721608 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:37.741581 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:37.751048 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 20:51:37.766036 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:37.874199 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:37.880661 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:37.964344 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:37.967084 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 20:51:38.092226 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 20:51:38.123200 18587 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:51:38.123409 18587 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:51:38.124040 18587 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:51:38.126160 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:38.126410 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:51:38.126505 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:51:38.126681 18587 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:38.209046 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a906569ab3b14520b4876bc8d5450f7d: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:38.293164 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 20:51:38.302201 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:38.335007 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:38.338178 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:38.394819 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:51:38.425531 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 20:51:38.450922 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 20:51:38.732687 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:39.049333 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:39.073863 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:39.162616 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:39.264925 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:39.267174 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d1b9ff3c921d474fa5ad106d1c17ab00: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:39.305264 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:39.312881 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:39.320979 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:39.403483 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:51:39.562629 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:39.734210 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240502 20:51:39.735443 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:39.841356 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:39.865772 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:39.892624 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 20:51:39.954860 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 20:51:39.959640 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:39.992319 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 20:51:40.059005 19696 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:40.059213 19696 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:51:40.059808 19696 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:51:40.060447 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:51:40.060653 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:51:40.061149 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:40.061849 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:40.061916 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:51:40.062184 19696 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:40.186260 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:40.219936 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:40.320418 26796 catalog_manager.cc:4573] TS 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 84484d65e2bb4e23ac621a39434ce382: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:40.366262 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:40.379734 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:51:40.397197 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:40.404609 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:40.515578 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:40.532243 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:40.614285 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:40.769461 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 20:51:40.795512 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 20:51:40.886447 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:40.907487 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:40.988106 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 20:51:41.005704 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 20:51:41.040150 19798 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:51:41.040414 19798 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:41.041255 19798 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:51:41.041908 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:41.042366 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:51:41.042790 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:41.043025 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:51:41.043259 19798 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:51:41.043916 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:41.069448 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:41.256575 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:41.340169 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 20:51:41.456712 19798 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:51:41.456887 19798 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:51:41.457430 19798 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:51:41.458580 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:51:41.458664 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:41.458761 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:51:41.458931 19798 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:41.502743 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:41.513685 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:41.571813 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:41.715623 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:41.752194 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:41.793144 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 20:51:41.863575 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:41.936682 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:41.992837 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:51:42.005906 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:51:42.217655 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:42.269675 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:42.578426 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 20:51:42.588762 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:42.608456 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:42.628325 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fdd2311ec0b645f5a79098a32c6deba2: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:42.708868 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:42.741000 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:42.802716 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:42.891762 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:42.970548 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:43.007853 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 20:51:43.074257 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:43.118523 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 20:51:43.147212 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 20:51:43.154023 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:43.245303 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:43.354153 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:43.359021 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:43.375735 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 20:51:43.441107 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:43.538343 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:43.575402 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 20:51:43.669411 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f2dfc0bbbbca4cc28a97480007343424: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:43.859365 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:44.063956 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:44.100746 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:44.105317 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 20:51:44.207927 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:44.290119 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:44.359990 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:44.412308 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:44.568087 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:44.666196 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:51:44.687687 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:44.747701 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 20:51:44.760600 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:44.995608 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:45.004173 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 20:51:45.008806 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:45.209995 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:45.248538 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:45.315503 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:45.332420 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:45.341168 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:45.422609 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 20:51:45.431305 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:45.459481 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:45.526839 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:45.597635 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:45.598654 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 20:51:45.631633 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:45.694015 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:45.705857 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:45.858556 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:45.953106 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:51:45.968421 20239 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:45.968654 20239 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:51:45.969321 20239 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:51:45.969735 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:51:45.969885 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:51:45.970777 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:45.970957 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:45.971021 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:51:45.971191 20239 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:46.066993 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:46.080672 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:46.412935 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:46.450881 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:46.490952 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:46.494627 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:46.599855 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:46.644997 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:46.742503 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:46.846787 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:46.912827 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:47.008774 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:47.124742 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:51:47.201885 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:47.251150 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:47.693558 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:47.737995 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:47.751614 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:47.879982 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:47.937559 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:47.960193 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:47.960328 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:48.071323 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:51:48.124992 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:48.135383 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:48.166441 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 20:51:48.185827 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:48.209659 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 20:51:48.210556 20503 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:48.210748 20503 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:51:48.211532 20503 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:51:48.211966 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:48.212181 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:51:48.212965 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:48.213132 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:48.213192 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:51:48.213440 20503 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:48.214591 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:48.308769 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:48.358150 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:48.481858 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:48.555589 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 20:51:48.556921 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:48.581425 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:48.795768 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:48.799579 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:48.803349 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:48.978598 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:49.029597 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:49.047027 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:49.066658 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:49.082844 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:49.271785 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:49.404677 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:49.492857 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:49.707033 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:49.715086 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:49.720682 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:49.760283 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 20:51:49.946139 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:51:50.018555 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:51:50.212877 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:50.264350 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:50.332916 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:50.336313 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:50.407491 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:50.413157 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:50.486649 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:50.499951 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:51:50.500367 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:50.520507 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:50.550176 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:50.572806 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:50.583748 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:50.687333 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:50.800685 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:50.825628 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 20:51:50.841651 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fdd2311ec0b645f5a79098a32c6deba2: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:51:50.904578 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:51.045364 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:51.188638 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:51.202190 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:51.317979 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:51.329836 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:51.359669 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:51.362565 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:51.560022 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:51.628455 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:51.755601 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:51.880789 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f2dfc0bbbbca4cc28a97480007343424: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:51.909440 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:52.059505 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:52.270582 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:52.367561 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:51:52.451125 20504 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:52.451357 20504 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:51:52.452140 20504 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:51:52.452566 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:51:52.452757 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:51:52.453567 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:52.453704 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:52.453780 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:51:52.453923 20504 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:52.466415 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:52.632848 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:51:52.717315 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:52.761521 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:52.862119 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:52.977170 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:52.986164 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:53.000157 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:53.065435 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:53.079916 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:53.106748 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:53.112027 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:53.122069 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:53.179845 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 20:51:53.204855 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:53.245376 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:53.275911 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:53.487774 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:53.505277 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 20:51:53.512315 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:53.580286 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:53.719120 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:53.828931 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:53.898183 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:53.916920 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 20:51:53.981799 20805 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:53.981961 20805 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:51:53.982478 20805 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:51:53.983029 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:51:53.983232 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:51:53.984110 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:51:53.984203 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:53.984249 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:51:53.984400 20805 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:54.007670 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:54.161976 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:54.378604 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:54.401981 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:54.666177 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:54.764701 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 20:51:55.020251 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:51:55.021449 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:55.067981 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:51:55.095149 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:55.236056 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:55.253710 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:55.257740 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:55.343251 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:55.355170 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:55.357900 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:55.421454 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:55.423918 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:55.486515 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:55.493245 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:55.507416 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:51:55.508268 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:55.720937 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:55.746769 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:55.763010 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:55.801908 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:55.891172 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 20:51:55.957396 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:56.030799 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:56.264822 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:56.308497 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:56.412766 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:56.456532 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:56.518858 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:56.593827 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:56.627094 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:51:56.792121 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:51:56.953505 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:57.178711 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:51:57.438270 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:51:57.523243 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:57.600723 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:57.693804 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:51:57.701841 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:51:57.751381 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:57.776854 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:57.783999 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:57.861173 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:57.951027 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:51:57.981191 21429 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:51:57.981388 21429 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:51:57.982107 21429 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:51:57.982446 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:51:57.983048 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:51:57.983310 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:51:57.983634 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:51:57.983808 21429 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:51:57.984118 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:58.036871 21323 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:51:58.037051 21323 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:51:58.037631 21323 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:51:58.038208 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:51:58.038406 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:51:58.039150 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:51:58.039247 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:51:58.039337 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:51:58.039517 21323 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:51:58.060463 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:58.164901 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:58.229718 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:51:58.250020 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:51:58.256145 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:58.258396 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:58.327916 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 20:51:58.330581 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:51:58.347214 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 20:51:58.400956 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:58.708556 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:51:58.776043 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:51:58.800623 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:58.840131 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:51:58.896828 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:51:58.995545 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:51:59.108474 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:51:59.177680 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:51:59.269798 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:51:59.337663 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:51:59.361889 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:51:59.547312 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:51:59.710407 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:51:59.766842 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 20:51:59.789600 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:52:00.077956 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 20:52:00.078740 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:00.222637 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:52:00.294616 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:52:00.337625 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:00.377367 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:00.449662 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:00.490921 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:52:00.564585 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 20:52:00.565690 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:00.605633 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:00.641043 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:00.698251 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:52:00.740100 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 20:52:00.862605 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:00.897286 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:52:00.963721 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:01.029778 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:52:01.148998 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:52:01.160316 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 20:52:01.254359 21429 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:52:01.254514 21429 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:52:01.255048 21429 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:52:01.256414 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:52:01.258627 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:01.258745 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:52:01.258957 21429 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:01.292120 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:52:01.318473 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:01.357801 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:01.620509 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:01.640379 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:01.653795 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:52:01.683135 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:01.846148 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:01.861192 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:01.875205 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:52:02.102864 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:02.159242 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:02.260713 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:02.558252 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:02.740494 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:52:02.806793 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 20:52:02.880612 21429 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:52:02.880831 21429 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:52:02.881455 21429 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:52:02.882670 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:52:02.882794 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:52:02.882858 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:52:02.883024 21429 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:02.899688 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:02.900796 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:02.929699 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 20:52:02.931763 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:52:02.948529 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:03.197950 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:03.212936 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:03.312953 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:03.319552 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:03.362329 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:03.377393 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:52:03.446714 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:03.592473 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:03.615451 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:52:03.645983 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:52:03.729555 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:03.860348 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:03.879010 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 20:52:03.881458 21964 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:03.881613 21964 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:52:03.882223 21964 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:52:03.882632 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:52:03.882812 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:52:03.883347 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:03.883533 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:03.883569 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:52:03.883750 21964 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:52:03.994544 21971 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:03.994774 21971 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:52:03.995571 21971 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:52:03.996250 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:52:03.996479 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:52:03.997115 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:52:03.997488 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:52:03.997582 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:52:03.997781 21971 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:04.009542 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:04.048662 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:04.186502 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:04.227308 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:04.230473 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:04.239199 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:52:04.422307 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:04.520690 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:04.604885 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:04.735954 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:04.790704 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240502 20:52:04.873697 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:04.957966 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:04.963088 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:52:04.963973 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:05.004432 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:05.099013 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:05.170380 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:52:05.200248 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:52:05.233984 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:05.332816 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:05.550899 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 20:52:05.618602 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:05.654177 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240502 20:52:05.731926 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:05.825026 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:05.842190 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:05.865797 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:05.965664 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:05.993024 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:06.021732 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:06.027760 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:06.080183 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:06.132364 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:06.227133 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:06.369874 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:06.478875 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:06.532763 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:06.534529 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:06.594610 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:06.611116 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:06.674224 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:06.689618 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:06.864058 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:07.041189 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:07.187381 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:07.264364 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fdd2311ec0b645f5a79098a32c6deba2: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:52:07.342588 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:07.366775 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:52:07.388418 22041 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:52:07.388597 22041 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:52:07.389169 22041 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:52:07.390573 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:52:07.391019 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:07.391093 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:52:07.391350 22041 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:07.459879 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:07.564985 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:52:07.577719 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:07.695045 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:07.890043 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:07.913676 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:08.031024 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:08.189332 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 20:52:08.238516 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:08.314574 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f2dfc0bbbbca4cc28a97480007343424: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:52:08.359499 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:08.473464 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:08.535733 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:08.555549 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:08.561641 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:08.575496 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:08.598150 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:08.601310 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:08.624619 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:08.668926 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:52:08.841679 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:08.899065 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:08.939736 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:08.949393 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:08.952742 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:08.975821 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:09.163137 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:09.188264 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:09.206985 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:09.522264 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:09.784063 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:52:09.853222 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:52:09.880069 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:09.912710 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:10.000344 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:10.005229 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:10.071125 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:10.107380 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:52:10.171597 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:10.291921 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:52:10.372917 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 20:52:10.413798 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:10.505824 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:10.599812 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:10.677893 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:10.694217 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:10.746309 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:52:10.828661 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:10.855568 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:10.954499 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:11.068784 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:11.180042 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:11.206401 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:11.239687 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:11.263355 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:11.351557 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:11.363274 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:11.381714 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:11.408156 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:11.510969 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:11.584218 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:11.677739 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:11.732403 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:11.804932 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:11.980288 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:12.116216 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:12.119506 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:12.485675 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:12.604943 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:12.612329 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:12.629632 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:52:12.724933 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:12.819651 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 20:52:12.915274 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:13.081374 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:13.230428 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:13.318595 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:13.328907 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:13.378974 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:13.422076 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:13.613458 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:13.709985 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:13.745225 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:13.756845 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:13.761359 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:13.783030 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 20:52:13.787827 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:13.879905 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:13.891531 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:13.991641 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:13.998179 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:14.001293 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:14.152045 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:14.164160 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:14.334674 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 20:52:14.338407 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:14.376204 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:14.636489 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:14.698463 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:14.829031 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240502 20:52:14.838784 21323 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:14.839006 21323 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:52:14.839732 21323 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:52:14.840322 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:52:14.840557 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:52:14.840731 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:52:14.841132 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:52:14.841210 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:52:14.841416 21323 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:14.882624 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 20:52:14.971447 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:15.050941 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:52:15.103142 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:15.251653 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:52:15.338004 23039 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:52:15.338198 23039 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:52:15.339061 23039 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:52:15.339375 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:52:15.340113 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:52:15.340529 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:15.340737 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:52:15.340947 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:52:15.341033 23039 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:52:15.370241 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:52:15.371098 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:15.400264 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 20:52:15.569500 23039 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:15.569655 23039 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:52:15.570302 23039 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:52:15.570736 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:52:15.570894 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:52:15.571795 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:15.571969 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:15.572033 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:52:15.572222 23039 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:15.622979 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:15.646553 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:15.740159 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:15.752830 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 20:52:15.753710 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:15.805614 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:16.175025 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:16.219202 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:16.261116 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:16.297151 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:16.350525 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:16.386276 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:16.438809 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:16.441128 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:16.487169 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:16.558992 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:16.573689 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:16.584591 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:16.696363 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:16.753891 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:16.779881 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:16.978602 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:17.118773 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:17.220448 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:17.260226 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:17.307114 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:52:17.388622 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:17.501986 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:17.611943 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:17.687785 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 20:52:17.812173 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:17.873059 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:18.011304 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:18.032661 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:18.166631 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:18.185374 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:18.278373 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:18.350533 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:18.528918 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:18.533545 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:18.747838 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:18.921241 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240502 20:52:18.922066 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:18.922201 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:18.951988 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:18.980780 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:19.011137 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:19.023120 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:19.053011 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:19.060753 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:19.068408 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:19.099864 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:19.537635 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:52:19.561856 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:19.566460 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:19.582170 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:19.854360 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:19.907819 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 20:52:19.908790 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:19.963284 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:19.976227 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:20.073164 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240502 20:52:20.144605 23386 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:52:20.144788 23386 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:52:20.145299 23386 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:52:20.146464 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:52:20.146672 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:52:20.146790 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:52:20.146972 23386 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:20.166834 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 20:52:20.258756 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:20.397015 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:20.412302 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:52:20.413378 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:20.497525 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:20.508415 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:20.662904 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:20.757447 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:52:20.791178 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:20.835624 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:20.921355 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:21.035851 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:21.206844 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:21.214342 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:21.226697 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:21.261363 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:21.363435 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:21.393647 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:21.486850 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:21.524710 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:21.588896 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:21.672827 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:21.727291 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:21.870497 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:21.977229 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:52:22.118710 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:22.229259 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:22.324324 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:22.330076 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:22.368419 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:22.494812 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:22.595330 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 20:52:22.638804 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:22.660279 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:22.897251 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:22.927150 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:22.929836 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:23.065819 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:23.228163 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:23.232425 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:23.341029 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:23.455369 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:23.467849 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:23.558662 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:23.582479 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:23.792526 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:23.905761 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:23.986405 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:24.055896 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:52:24.056951 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:24.117347 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:24.126559 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:24.142258 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:24.298883 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:24.314306 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:24.495134 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:24.543095 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:24.592836 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:52:24.695109 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:24.848984 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:24.874652 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:24.974296 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 20:52:24.975296 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:25.010372 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 20:52:25.138196 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:25.185642 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:25.225639 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:25.324522 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:25.464316 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:52:25.465174 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:25.576812 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:25.596150 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:25.607731 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:25.640758 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:25.674146 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:25.850172 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:52:25.937047 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:25.960289 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:26.131554 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:26.211760 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:26.279588 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:26.369426 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:26.394929 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:26.444146 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:26.556576 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:26.621619 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:26.752666 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:26.775854 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:26.858213 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:26.957166 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:27.135901 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:27.210198 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:27.228966 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:27.267470 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:52:27.376709 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:27.445466 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:27.586967 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:27.593397 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 20:52:27.695063 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:27.797245 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:27.867465 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:27.941891 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:28.010490 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:28.014500 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:28.074724 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:28.075001 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:28.144014 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:28.378222 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:28.389955 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:28.507716 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:28.691773 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:28.697731 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:52:28.796749 24132 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:28.796936 24132 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:52:28.797660 24132 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:52:28.798070 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:52:28.798323 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:52:28.799315 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:52:28.799439 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:52:28.799541 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:52:28.799741 24132 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:28.882784 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:28.921878 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:29.024642 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 20:52:29.105931 24242 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:29.106185 24242 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:52:29.107095 24242 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:52:29.107977 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:52:29.108238 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:52:29.108690 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:29.109447 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:29.109531 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:52:29.109735 24242 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:29.143602 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:29.206856 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:29.235757 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240502 20:52:29.280237 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:29.283296 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:29.362409 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:29.473569 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:29.500038 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:29.583141 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:29.586100 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:29.769193 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:29.781351 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:29.936427 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:29.954468 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:52:30.025616 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240502 20:52:30.044749 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:30.091274 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 20:52:30.250183 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:30.270615 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:30.317759 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:30.460052 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:30.524626 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:30.616916 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:30.639612 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:52:30.663699 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:30.715675 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:30.876698 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240502 20:52:30.932381 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:31.028497 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:31.097805 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:31.178148 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 20:52:31.339949 24504 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:31.340148 24504 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:52:31.340824 24504 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:52:31.341547 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:52:31.341827 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:52:31.342476 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:52:31.342725 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:31.342996 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:52:31.343070 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:52:31.343266 24504 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:31.378923 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:31.442553 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:31.522233 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:31.575688 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:31.585567 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:31.637624 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:31.771392 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:31.859306 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:31.912065 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:31.978320 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:31.995507 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:32.094797 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:32.138744 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:32.292048 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:52:32.319000 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:32.336429 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:32.472360 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:32.618520 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 20:52:32.631120 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:32.650627 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:32.910463 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:32.947746 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:33.083722 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:33.122318 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:33.169018 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:33.252974 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:33.330111 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:33.363301 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:33.624293 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:33.800748 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 20:52:33.824152 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:33.879351 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:52:33.987749 24695 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:52:33.987937 24695 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:52:33.988765 24695 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:52:33.989665 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:52:33.989660 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:52:33.990821 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:52:33.991092 27298 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:52:33.991317 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:33.991333 24696 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:52:34.137403 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:34.228771 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:34.244385 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:34.247349 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:34.253895 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:34.272193 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 20:52:34.293476 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:34.431845 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:34.440718 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:34.499186 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:34.514536 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:34.534276 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:34.728897 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:34.814378 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:52:34.911049 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:34.925606 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:34.946219 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:35.099462 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 20:52:35.100275 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:35.208474 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:35.276667 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:35.420037 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240502 20:52:35.531313 24696 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:52:35.531514 24696 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:52:35.532119 24696 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:52:35.533605 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:52:35.533855 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:35.533927 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:52:35.534144 24696 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:35.578737 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:35.655838 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:52:35.669183 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 20:52:35.728931 24696 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:35.729142 24696 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:52:35.729748 24696 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:52:35.730206 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:52:35.730386 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:52:35.731989 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:35.732510 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:35.732605 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:52:35.732793 24696 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:35.791226 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:35.840838 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:35.881378 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 20:52:35.885537 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:35.914880 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:35.980243 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:36.082541 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:36.158007 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:36.312675 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:36.413735 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:36.493085 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:36.593571 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:36.617913 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:36.680572 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:36.757897 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:36.828956 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:36.861850 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 20:52:36.973683 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:37.065619 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:37.081797 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 20:52:37.092695 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:37.105034 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:37.163509 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:37.495507 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:37.552892 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:52:37.553273 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240502 20:52:37.615518 24969 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:37.615687 24969 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:52:37.616240 24969 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:52:37.616660 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:52:37.616920 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:52:37.617969 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:37.618126 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:37.618189 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:52:37.618381 24969 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:37.655941 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:37.769287 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:37.778442 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:37.782075 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:52:37.782943 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:38.069660 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:38.081885 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:38.176864 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 20:52:38.200794 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:38.245492 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:38.279129 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:38.403654 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:38.485374 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 20:52:38.535696 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:38.663676 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:38.899011 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:38.945812 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:38.948453 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 20:52:38.982924 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:39.037551 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:39.092829 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:39.111829 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:39.330891 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:39.399823 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:39.412015 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 20:52:39.477385 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 20:52:39.490589 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 20:52:39.494971 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:39.516098 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:39.604250 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 20:52:39.736063 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:39.793162 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:40.045507 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fdd2311ec0b645f5a79098a32c6deba2: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:52:40.077239 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 20:52:40.085701 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:40.096228 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:40.106143 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:40.113312 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 20:52:40.114188 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:52:40.232270 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 20:52:40.339910 25316 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:40.340397 25316 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:52:40.341540 25316 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:52:40.341885 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:52:40.342070 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:52:40.342495 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:52:40.342996 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:52:40.343079 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:52:40.343225 25076 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:40.434238 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:40.444265 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:40.629966 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 20:52:40.637296 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:40.640538 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:40.790205 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:52:40.832269 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:40.900583 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:52:40.940299 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:41.104063 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:52:41.135052 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 20:52:41.147926 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:41.179138 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:41.449951 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:41.492802 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:41.549430 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:41.608389 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 20:52:41.640566 25315 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:41.640790 25315 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:52:41.641476 25315 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:52:41.642007 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:52:41.642203 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:52:41.642990 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:41.643149 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:41.643208 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:52:41.643396 25315 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:41.653240 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:41.685248 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:41.721817 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:41.908622 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:41.971629 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:42.001698 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:42.075112 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:42.117717 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:42.155179 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:42.164602 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:42.315140 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:42.438565 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:42.483280 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:52:42.678342 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:42.693478 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:42.717332 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:42.733584 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:42.747965 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:42.840068 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:43.003475 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 20:52:43.128221 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:43.310580 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:43.366708 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:43.459316 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:43.604863 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:43.658352 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:43.685015 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:43.791456 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:43.834944 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:43.840430 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:43.845748 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:43.891032 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:43.968374 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:44.267025 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:44.447109 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:44.521672 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 20:52:44.533327 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:44.588042 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:44.707252 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:44.752445 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:44.796521 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:44.901657 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:44.940378 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:44.956030 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:45.035589 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:45.117369 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 20:52:45.167212 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:45.216540 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:52:45.244232 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:45.289903 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:45.297242 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 20:52:45.399765 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:45.445890 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:45.487363 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:45.783061 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:45.795029 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:52:45.876382 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:45.908632 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:52:46.044807 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:46.200304 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:46.239187 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:46.267256 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:46.291711 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:46.377630 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:46.436187 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:46.458700 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:46.481765 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:46.650538 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:46.816980 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:46.818136 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:46.838454 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:46.893499 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:47.035229 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:47.260735 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:47.279867 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:47.344992 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:47.363843 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:47.429430 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:47.432755 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:47.583050 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:52:47.595304 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:47.727814 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 20:52:47.734467 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:47.877660 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:48.019579 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:48.024886 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:48.149703 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:48.242453 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:48.439452 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:48.529680 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:48.666808 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:48.707803 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:48.730671 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:48.830518 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:48.891691 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:48.948076 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:48.959417 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:49.009125 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:49.175448 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:49.182086 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:49.313848 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:49.412992 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:49.516652 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:49.604276 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:52:49.722465 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:49.766599 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:49.785539 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:49.828130 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:49.828714 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:49.903283 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:49.961045 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 20:52:50.064029 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:50.067764 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:52:50.072484 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:50.130232 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:52:50.216187 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:50.235083 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:50.386927 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:50.445894 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 20:52:50.486445 26166 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:50.486629 26166 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:52:50.487350 26166 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:52:50.487933 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:52:50.488188 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:52:50.489338 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:52:50.489442 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:50.489593 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:52:50.489800 26166 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:50.502705 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:50.803536 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:52:50.878508 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:50.890404 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:51.025867 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:52:51.034700 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:51.046650 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:51.163313 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:51.219100 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:51.225886 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:51.325721 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:51.337323 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:51.439307 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:51.607542 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:51.620020 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:51.878546 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:51.939646 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:51.947937 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:51.955538 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:51.973563 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:52.133795 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:52.241030 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:52.294082 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:52.334789 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:52.375186 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240502 20:52:52.402709 26386 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:52:52.402928 26386 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:52:52.403697 26386 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:52:52.403985 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:52:52.404314 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:52:52.405045 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:52.405141 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:52:52.405373 26386 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:52:52.405974 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:52:52.455257 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 20:52:52.487210 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:52.563150 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:52.597451 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:52.642814 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:52:52.649801 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:52.661737 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:52.992338 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:53.150700 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:53.220547 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:53.237326 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 20:52:53.378226 26386 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:52:53.378392 26386 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:52:53.378892 26386 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:52:53.380138 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:52:53.380282 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:53.380411 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:52:53.380594 26386 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:53.431329 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:53.547616 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:53.562589 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:53.690873 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:53.870518 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:53.896008 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:53.923776 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:53.941335 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 20:52:54.082685 26590 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:54.082867 26590 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:52:54.083561 26590 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:52:54.084563 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:52:54.084784 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:52:54.084980 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:52:54.085186 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:54.085263 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:52:54.085469 26590 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:54.194972 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:54.365918 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:54.369868 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:54.467257 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:54.477372 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:54.605237 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:54.607467 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240502 20:52:54.647445 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:54.788560 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:54.824187 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:54.852125 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:54.918884 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:54.935416 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:54.989578 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 20:52:54.990850 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:55.038414 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:52:55.162039 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 20:52:55.162950 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:55.191574 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:55.214649 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:55.320341 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:55.357900 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:55.537732 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:55.751608 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:55.810494 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:55.853340 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 20:52:55.854151 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:55.883070 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:56.093384 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:56.137288 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:56.141845 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 20:52:56.214144 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:56.379032 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:56.556557 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:56.613381 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:56.659497 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:56.773372 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:56.828909 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:56.868909 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:56.880874 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240502 20:52:56.937711 26320 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:52:56.938161 26320 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:52:56.938983 26320 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:52:56.939522 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:52:56.939754 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:52:56.940379 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:52:56.940429 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:56.940513 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:52:56.940572 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:52:56.940742 26320 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:52:57.218784 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:57.238961 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:57.263998 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 20:52:57.320797 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:57.387849 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:57.452971 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:57.455408 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:57.470710 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:52:57.477844 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:57.578323 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:52:57.649930 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:57.738276 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:57.785408 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:57.805683 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:57.908828 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:58.134274 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:58.273528 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:58.307015 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:58.335175 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:52:58.475410 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:58.682886 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:58.820323 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:58.848557 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:58.922216 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:59.103396 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:59.204474 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:52:59.219010 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:52:59.311833 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:59.333006 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:52:59.494175 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:52:59.597995 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:52:59.627858 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:59.704627 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:52:59.724334 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:59.734551 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:52:59.797559 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:52:59.857458 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:59.866526 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 20:52:59.930995 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:59.932168 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:52:59.973663 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:53:00.070704 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:00.108578 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:00.187608 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 20:53:00.233456 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:00.264305 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:53:00.355942 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:00.434763 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:53:00.447607 27692 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:53:00.447837 27692 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:53:00.448598 27692 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:53:00.448994 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:53:00.449798 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:53:00.450054 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:53:00.450383 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:53:00.450501 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 1 yes votes; 4 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:53:00.450699 27692 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:53:00.565506 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:00.667246 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:53:00.698652 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:00.853413 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 20:53:00.946662 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:53:00.968999 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:01.154978 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 20:53:01.229504 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:53:01.336198 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:01.460758 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:01.541574 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:01.686362 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:01.746111 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:01.784551 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:01.821462 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:01.893572 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:02.021014 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:02.024916 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:02.118554 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:02.192893 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:02.231969 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:02.296041 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:02.321491 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 20:53:02.375994 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:02.389461 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:02.470892 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:02.535224 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:53:02.562618 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:53:02.643555 27736 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:02.643733 27736 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:53:02.644269 27736 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:53:02.644711 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:53:02.644893 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:53:02.646281 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:53:02.646448 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:02.646507 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:53:02.646723 27736 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:02.680778 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:02.693964 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:02.789448 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:02.867408 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:02.902635 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:02.916765 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:03.053916 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:03.268437 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:03.323884 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:03.332212 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:03.492820 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:03.946586 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:03.989506 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:04.023911 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:04.244884 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:04.280820 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:04.335594 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:04.340708 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:04.341977 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:04.415380 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:04.499815 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:04.708125 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240502 20:53:04.732867 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:04.737280 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:04.760422 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:04.876653 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:04.878746 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 20:53:04.928155 28102 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:04.928416 28102 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:53:04.929204 28102 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:53:04.929903 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:53:04.930162 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:53:04.930680 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:53:04.931485 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:53:04.931572 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:53:04.931742 28102 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:04.962798 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:04.987195 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:05.088172 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:53:05.096282 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:05.099263 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 20:53:05.104445 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:05.135108 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:05.195569 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 20:53:05.296569 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:05.382498 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:05.389672 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:05.440946 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:05.654449 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:05.684051 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:05.794641 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 20:53:05.826031 28189 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:05.826215 28189 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:53:05.826923 28189 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:53:05.827387 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:53:05.827570 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:53:05.828604 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:53:05.828722 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:05.828787 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:53:05.828943 28189 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:05.870808 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:05.897846 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:53:06.181705 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:06.254947 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 20:53:06.300711 28189 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:53:06.300897 28189 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:53:06.301474 28189 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:53:06.302817 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:53:06.303165 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:53:06.303362 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:53:06.303625 28189 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:06.394860 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:06.687389 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:06.764542 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:06.792726 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:06.825366 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:06.855422 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:06.932626 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:06.973371 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:07.075443 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:07.089373 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:07.124078 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:07.170099 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:07.194173 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:07.211987 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:07.290820 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:07.435336 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 20:53:07.493119 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:07.547485 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:07.582440 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:53:07.627156 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:07.718425 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:07.744911 28189 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:53:07.745169 28189 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:53:07.746107 28189 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:53:07.746482 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:53:07.746842 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:53:07.747614 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:53:07.747822 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:53:07.748019 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:07.748032 28189 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:53:07.772456 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:07.801884 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:07.833498 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:07.904227 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:07.971758 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:08.043640 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:08.054814 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:08.085798 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:08.243804 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:08.852258 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:08.871012 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:09.101882 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:09.173313 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:09.268280 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:09.290807 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:09.305444 28189 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:53:09.305603 28189 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:53:09.306311 28189 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:53:09.306643 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:53:09.307016 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:53:09.307677 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:09.307755 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:53:09.307945 28189 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:53:09.308427 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:53:09.316455 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:09.449826 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:09.450029 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:09.508882 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:09.514034 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:09.543435 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:09.706097 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:09.713698 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 20:53:09.720566 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:09.741863 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 20:53:09.775867 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:09.782325 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:09.808316 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:09.830705 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:53:09.926947 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:53:10.015712 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:10.183300 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:10.229228 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:10.231603 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:10.256667 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 20:53:10.302628 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:10.390010 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240502 20:53:10.398871 28296 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:10.399237 28296 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:53:10.399844 28296 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:53:10.400290 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:53:10.400456 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:53:10.401468 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:53:10.401562 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:10.401638 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:53:10.401794 28296 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:10.417917 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 20:53:10.423543 28638 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:10.423722 28638 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:53:10.424281 28638 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:53:10.424989 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:53:10.425284 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:53:10.426095 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:53:10.426240 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:10.426298 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:53:10.426499 28638 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:10.566295 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:10.623737 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:10.700824 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:10.754040 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:10.810328 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:10.915819 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:53:11.268924 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 20:53:11.367154 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:11.461444 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:11.474031 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:11.487298 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:11.676115 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:11.727047 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:11.841159 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:11.906673 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:11.966360 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:12.022022 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:12.059020 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:12.178114 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 20:53:12.184687 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:12.225168 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:12.261770 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:12.325140 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:12.395830 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:12.443197 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:12.485517 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:12.504555 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:12.544145 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:12.572597 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:53:12.730551 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:12.889406 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:12.991509 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:13.135496 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:13.192850 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:13.206274 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:13.318775 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 20:53:13.340778 28923 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:13.340951 28923 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:53:13.341599 28923 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:53:13.342087 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:53:13.342257 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:53:13.342751 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:53:13.342963 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:53:13.343029 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:53:13.343164 28923 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:13.375535 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:13.488298 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:13.687781 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:13.771238 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:13.838212 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:13.872426 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:14.034648 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:14.286675 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:14.291973 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:14.424901 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:14.480808 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:14.507519 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:14.648180 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:14.658704 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:14.708472 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:14.720433 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:14.726998 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:14.735285 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:14.807035 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 20:53:14.827600 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:14.874788 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240502 20:53:14.905055 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:14.981628 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:14.982231 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:15.205155 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:15.278863 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 20:53:15.317842 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:53:15.332132 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:15.454221 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:15.490857 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:15.641115 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:15.766731 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:15.812817 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:15.846336 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:15.939476 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:53:16.105895 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:16.112509 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:16.125003 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:16.303707 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:53:16.314257 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:16.336998 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:16.627766 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 20:53:16.709182 29050 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:16.709370 29050 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:53:16.710234 29050 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:53:16.710850 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:53:16.711084 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:53:16.712106 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:53:16.712255 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:16.712313 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:53:16.712610 29050 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:16.754262 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:16.851233 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:16.878661 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:16.904796 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:16.972144 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:17.066973 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:17.081779 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:17.121274 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:17.159526 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:17.189464 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 20:53:17.196552 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:17.263931 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:17.382354 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:17.430231 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:17.436769 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:17.481024 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:17.621189 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:17.672359 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:53:17.865481 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:17.911470 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:17.911621 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:17.988690 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:18.028759 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:18.248667 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:18.471776 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:18.475157 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:18.625885 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:18.800252 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:18.931934 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:19.000178 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:19.114187 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:19.130896 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:19.174760 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:19.230559 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:19.367565 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:19.530834 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:19.532470 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:19.607165 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:19.614281 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:19.661279 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 20:53:19.846359 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:19.850071 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:19.853955 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:19.877498 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:19.979480 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:53:20.026302 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:20.086700 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:20.127190 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:20.197027 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:20.228335 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:20.293285 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:53:20.302788 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:53:20.363252 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:20.375881 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:20.634202 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:20.868441 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:20.932211 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:20.962522 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:20.964954 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:53:21.077037 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:21.154165 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:21.254263 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:21.330149 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 20:53:21.419214 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:21.626510 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:21.627679 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:21.769191 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:21.780488 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:21.797839 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:21.957202 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:21.961792 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:21.993799 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:22.006160 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:22.024029 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 20:53:22.234946 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 20:53:22.271862 29427 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:53:22.272013 29427 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:53:22.272631 29427 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:53:22.272948 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:53:22.273372 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:53:22.274379 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:53:22.274456 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:53:22.274539 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:22.274601 29427 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:53:22.357590 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:22.400708 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:22.464083 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:22.480090 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 20:53:22.496835 29715 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:53:22.496978 29715 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:53:22.497437 29715 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:53:22.498654 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:53:22.498762 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:22.498817 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:53:22.498975 29427 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:22.514194 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:22.561210 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:22.565058 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:22.605245 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:22.675935 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:53:22.713779 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:22.719280 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:22.853382 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:23.208751 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:23.272040 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:23.288342 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:23.427019 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:23.581188 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:23.677404 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:23.945484 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:24.058444 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:24.129995 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:24.197176 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:24.229910 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:24.274714 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:24.318378 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:24.541496 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:24.569195 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240502 20:53:24.585121 29803 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:24.585299 29803 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:53:24.585911 29803 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:53:24.586158 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:53:24.586303 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:53:24.588059 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:53:24.588178 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:24.588225 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:53:24.588438 29803 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:24.588820 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:24.616730 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:24.623441 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:24.671777 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 20:53:24.708832 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:24.757843 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:24.787804 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:24.876595 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:24.903451 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:24.906725 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:25.052438 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:25.099009 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:53:25.197016 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:53:25.207515 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:25.272585 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:25.313823 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:25.330667 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 20:53:25.333482 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:25.706192 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:25.764441 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:25.935192 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:25.935604 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:25.975684 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:53:26.017038 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:26.023809 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:26.332988 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 20:53:26.437927 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:26.710165 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:26.730561 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:26.731145 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:26.820116 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 20:53:26.875087 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:26.898801 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:26.910595 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:26.972746 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:27.006379 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:27.035887 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:27.140348 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:27.196199 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:27.201341 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:27.226764 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:27.239256 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:27.393299 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:27.394500 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:27.429744 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:27.497831 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:27.521039 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:53:27.533630 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:27.662230 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:27.793123 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:27.861753 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:28.159329 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:28.216207 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240502 20:53:28.251546 30083 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:28.251796 30083 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:53:28.252524 30083 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:53:28.253101 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:53:28.253347 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:53:28.254284 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:53:28.254457 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:53:28.254515 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:53:28.254714 30083 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:28.284535 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:28.292088 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:28.293774 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:28.584251 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:28.928531 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 20:53:29.013513 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:29.033363 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:29.155274 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:29.180749 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 20:53:29.238160 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:29.240171 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:29.364045 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:29.610312 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:29.640872 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:29.668704 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 20:53:29.676477 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:29.702983 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:29.823791 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:29.838213 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:29.934198 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:29.948858 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:29.964473 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:30.000020 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:30.038827 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:30.048094 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:30.049693 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:30.113335 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:30.148475 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:53:30.230522 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 20:53:30.267798 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:30.392509 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 20:53:30.513509 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240502 20:53:30.665791 29966 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:53:30.666098 29966 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:53:30.666934 29966 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:53:30.667696 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:53:30.668040 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:53:30.668861 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:30.668956 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:53:30.669095 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:53:30.669241 29966 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:53:30.701205 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:30.815580 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:30.852089 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:30.862682 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:30.918748 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:30.982896 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 67 similar messages]
I20240502 20:53:31.254045 30083 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:31.254318 30083 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:53:31.254941 30083 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:53:31.255712 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:53:31.255941 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:53:31.256719 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:53:31.257112 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:31.257232 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:53:31.257405 30083 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:31.285614 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:31.422313 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 20:53:31.426038 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:53:31.563843 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:31.584069 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:53:31.717916 29966 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:31.718124 29966 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:53:31.718895 29966 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:53:31.719465 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:53:31.719694 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:53:31.721156 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:53:31.721316 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:31.721376 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:53:31.721572 29966 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:31.729137 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:31.741565 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 20:53:31.777163 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 20:53:31.855357 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240502 20:53:31.958403 26799 maintenance_manager.cc:419] P e0c668105a124f0eb7227390002cc15e: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033351
I20240502 20:53:31.962906 26798 maintenance_manager.cc:643] P e0c668105a124f0eb7227390002cc15e: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.004s	user 0.002s	sys 0.000s Metrics: {"bytes_written":3373,"delete_count":0,"lbm_write_time_us":170,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":106,"threads_started":1,"update_count":56,"wal-append.queue_time_us":268}
I20240502 20:53:31.963407 26799 maintenance_manager.cc:419] P e0c668105a124f0eb7227390002cc15e: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.208403
I20240502 20:53:31.972867 26798 maintenance_manager.cc:643] P e0c668105a124f0eb7227390002cc15e: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.009s	user 0.009s	sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":17693,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":367,"lbm_read_time_us":218,"lbm_reads_lt_1ms":17,"lbm_write_time_us":225,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":162,"threads_started":2,"update_count":56}
W20240502 20:53:31.990881 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 20:53:32.023895 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 20:53:32.104826 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:32.133852 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:32.171597 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:32.289325 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 20:53:32.319336 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:32.372617 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:32.443317 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:32.468281 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:32.516772 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:32.532215 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:32.569825 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:32.588254 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:32.612478 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:32.714069 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:53:32.732771 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:33.012814 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:33.242089 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:33.328195 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 20:53:33.496591 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:33.532107 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 20:53:33.562085 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:33.955405 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:34.035571 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:34.054029 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:34.105906 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:34.119364 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:34.195547 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 20:53:34.352177 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:34.432039 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:34.462661 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:34.512569 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:34.549135 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:34.581256 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:34.662096 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:34.696617 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:34.708009 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:34.860463 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:34.894146 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:34.928345 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:34.941522 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:34.970678 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:34.981504 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:35.078683 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:35.158372 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:35.238175 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:35.250130 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 20:53:35.392045 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:53:35.396629 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 20:53:35.526329 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:35.601527 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:35.863473 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:35.965054 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:36.146404 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:36.164136 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:53:36.295197 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:36.446621 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:36.493394 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:53:36.549809 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:36.617987 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:36.623373 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:36.652700 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:36.657382 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:36.703831 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:36.860558 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 20:53:36.904278 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:36.923272 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:37.039228 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:37.067801 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:37.184103 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:37.235527 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:37.255581 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:37.358835 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:37.455477 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:37.468873 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:37.479616 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:37.544682 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:37.565886 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 20:53:37.654693 31146 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:53:37.654938 31146 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:53:37.655606 31146 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:53:37.657202 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:53:37.657449 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:53:37.657544 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:53:37.657749 31146 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:37.669339 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:37.733129 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:37.747092 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:37.811399 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:53:37.812530 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:53:37.919284 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:37.977190 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:38.120457 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:53:38.322238 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:38.703954 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:38.711432 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:38.717824 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:38.782572 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:38.980829 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:53:39.066035 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:39.114984 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:39.161278 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:39.240056 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:39.252668 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:39.309445 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:39.351081 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:39.374140 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:39.415035 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:39.488382 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:39.508872 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 20:53:39.524091 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:39.649638 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:39.816764 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:39.898018 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:40.026717 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:40.048853 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fdd2311ec0b645f5a79098a32c6deba2: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:53:40.087985 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:40.203982 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:40.214406 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:40.253198 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:40.257531 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:53:40.264408 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:40.399448 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:40.403738 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 20:53:40.415594 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:40.506408 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:40.551322 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:40.590682 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:53:40.674032 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:41.107717 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:53:41.247697 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:53:41.269959 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:41.316365 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:41.381587 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:41.445701 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:41.491150 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:41.543710 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:53:41.552523 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:41.656747 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:41.750706 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:41.799088 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:41.877628 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:41.903309 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:42.020148 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:42.050838 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:42.075008 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:42.151108 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 20:53:42.172178 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:42.229488 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:42.270151 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:42.275436 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:42.299743 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:42.445762 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:42.465482 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:42.715160 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:42.762238 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:42.791810 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:42.868952 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:42.874397 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:53:42.905329 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:43.206743 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:43.223487 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:43.245724 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240502 20:53:43.315269 31080 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:43.315435 31080 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:53:43.316061 31080 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:53:43.316540 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:53:43.316740 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:53:43.317391 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:53:43.317742 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:43.317821 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:53:43.317984 31080 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:43.622239 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:43.770220 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:43.785282 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:44.043283 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:44.067353 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:44.100344 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:44.220351 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:44.303067 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:44.303998 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:44.381434 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:44.510627 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:44.666702 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:44.677227 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 20:53:44.742872 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:44.744143 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:44.798286 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:44.802522 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:44.859318 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:44.928342 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:44.968534 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:53:45.009158 31807 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:45.009327 31807 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:53:45.009876 31807 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:53:45.010516 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:53:45.010702 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:53:45.011468 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:53:45.011638 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:45.011809 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:53:45.012096 31807 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:45.117029 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:45.145430 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:45.154969 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:45.234477 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:45.291492 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:45.304553 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:45.386953 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 20:53:45.396997 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:45.414376 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 20:53:45.543004 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:53:45.610546 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:45.772418 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:45.859107 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:46.117437 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:46.308756 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 20:53:46.318331 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:46.327486 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:46.656167 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:46.660116 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:53:46.692545 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:46.818612 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:46.891844 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:46.912294 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:46.973577 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:47.012285 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:47.012672 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:47.041994 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:47.140928 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:47.168838 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:47.179433 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 20:53:47.341960 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:47.345017 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:47.362093 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:47.408982 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:47.415683 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:47.612764 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:47.719801 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:47.747622 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:47.810302 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:47.819870 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:47.856174 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:47.908003 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:48.023468 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:53:48.302377 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:48.314725 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:48.388746 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 20:53:48.733418 31806 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:48.733618 31806 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:53:48.734274 31806 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:53:48.734525 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:53:48.734848 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:53:48.735615 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:48.737198 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:53:48.737346 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:48.737401 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:53:48.737553 31806 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:48.822906 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:48.847885 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:53:48.868340 31807 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:53:48.868520 31807 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:53:48.869302 31807 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:53:48.869845 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:53:48.870160 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:53:48.870680 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:48.870766 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:53:48.870841 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:53:48.870929 31807 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:53:48.942154 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:49.141883 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:49.308025 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:49.362108 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:49.420099 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:49.500092 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:49.519395 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:49.571807 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:49.685467 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 20:53:49.752785 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:49.757532 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:49.866757 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:49.893172 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:49.893173 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:49.989156 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:50.011268 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:50.064837 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:50.104647 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:50.114434 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:50.161087 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:50.192770 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:50.251125 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:50.293414 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:50.435230 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:53:50.437721 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:53:50.459856 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:53:50.498626 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:50.672953 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:50.701869 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:51.050874 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:51.301294 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:51.321907 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:51.337905 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:51.394337 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:51.414932 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 20:53:51.444731 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:51.614806 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:51.633477 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:51.680914 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:53:51.947569 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:52.066550 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:52.068409 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:52.092684 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 20:53:52.111168 31807 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:52.111382 31807 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:53:52.111991 31807 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:53:52.112891 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:53:52.113129 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:53:52.113641 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:53:52.113857 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:52.113924 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:53:52.114106 31807 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:52.122762 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:52.235425 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:52.238480 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240502 20:53:52.296070 31807 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:53:52.296265 31807 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:53:52.296783 31807 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:53:52.298241 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:53:52.298357 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:52.298431 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:53:52.298804 31807 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:52.424311 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:52.482270 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 20:53:52.487342 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:52.528774 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:52.562115 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:52.569836 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:52.617825 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:52.667403 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:53:52.697252 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:52.708185 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:52.708294 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:52.724187 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:52.764376 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:52.944592 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:52.980533 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:53.137915 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:53.553860 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:53.781658 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:53.828899 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:53.905297 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:54.016815 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:54.035280 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:54.244071 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:54.294013 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:54.346361 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:54.422801 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:54.477362 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:54.490963 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:54.541389 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:54.559386 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:54.766230 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 20:53:54.782774 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:54.873870 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:54.882097 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:54.914302 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:55.043996 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:55.055987 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:55.118662 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:55.119468 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:55.163127 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:55.174502 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:55.193128 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:55.196458 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:55.323763 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:53:55.395295 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:55.458173 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 20:53:55.469753 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 139 similar messages]
I20240502 20:53:55.513481 31807 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:53:55.513633 31807 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:53:55.514195 31807 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:53:55.514215 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:55.515568 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:53:55.515671 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:55.515725 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:53:55.515895 31807 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:55.575083 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240502 20:53:55.930444 32409 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:53:55.930641 32409 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:53:55.931291 32409 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:53:55.931912 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:53:55.932101 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:53:55.932586 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:53:55.932893 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:53:55.932967 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:53:55.933167 32409 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:53:55.966840 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:56.228639 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:56.370643 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:56.439414 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:53:56.466157 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:56.564266 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:53:56.614526 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:56.635294 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:56.645798 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:56.721642 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240502 20:53:56.809607 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:56.871706 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:56.989758 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:53:57.109181 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:57.204432 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:53:57.214335 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:53:57.216470 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:57.234423 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:57.354107 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 20:53:57.358810 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:57.403952 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:57.490594 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:57.503938 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:57.696784 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:57.756284 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:53:57.771391 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:57.791352 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:57.831759 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:53:57.837500 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:57.947314 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:53:57.966446 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:53:58.040807 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:58.184702 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:58.349172 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:58.449051 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:53:58.839329 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:53:59.059417 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:59.081728 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:53:59.176937 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:59.229308 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:53:59.275070 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:53:59.383101 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:53:59.506953 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:53:59.523193 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:53:59.531836 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:53:59.612323 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:53:59.711898 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 20:53:59.743953 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:53:59.753695 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:53:59.834707 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:53:59.861986 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:53:59.912127 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:53:59.988338 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:00.099388 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:00.156642 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:00.176803 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:00.224436 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:54:00.236816 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:00.253182 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:54:00.319335 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:54:00.354492 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:00.382508 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:00.502835 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:54:00.609778 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:00.726703 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:54:00.750418 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240502 20:54:00.922086 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:54:01.077728 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:54:01.204880   748 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:54:01.205139   748 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:54:01.205870   748 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:54:01.206360 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:01.206630 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:54:01.207507 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:01.207620 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:54:01.207731 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:01.207808   748 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:54:01.289000 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:54:01.426024 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:01.626010 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 20:54:01.701817   856 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:01.702162   856 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:54:01.703372 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:54:01.703609 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
I20240502 20:54:01.703713   856 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
W20240502 20:54:01.704460 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:54:01.704608 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:01.704701 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:54:01.704859   856 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:01.736704 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:01.766992 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:01.794268 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:54:01.878064 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:01.882381 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240502 20:54:01.903904   856 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:54:01.904078   856 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:54:01.904556   856 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:54:01.905759 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:54:01.905870 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:01.905965 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:54:01.906131   856 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:01.908320 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:01.945794 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:02.100109 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:54:02.152743 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:02.154516 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:02.170902 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:02.331117 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:02.339746 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:54:02.340350 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 20:54:02.347656 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:02.366962 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:02.408463 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:54:02.511376 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:02.529804 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:02.702450 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:02.766263 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:02.772394 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:54:02.874709 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:02.874884 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:02.968995 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:54:02.973420 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:03.135824 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:54:03.202441 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:03.324707 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:03.693418 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:03.832005 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:03.832764 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:04.155762 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:04.161600 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:04.254601 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:04.383481 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:04.412120 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:04.464323 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:04.511930 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:04.588125 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:04.632367 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:04.711344 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 20:54:04.768077 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:04.781337 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:04.825227 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:04.857244 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:04.875208 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:04.898519 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:04.996122 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:54:04.997244 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:05.072985 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:05.096504 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:05.112035 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 20:54:05.150694  1063 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:05.150875  1063 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:54:05.151644  1063 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:54:05.152073 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:05.152276 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:54:05.152963 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:05.153146 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:05.153209 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:54:05.153405  1063 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:05.349687 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:05.358559 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:05.434120 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:05.523800 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:54:05.625885 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:05.662416 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:05.955319 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:05.960908 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:54:06.118068 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:06.221344 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:06.256065 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240502 20:54:06.348969  1168 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:06.349203  1168 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:54:06.349907  1168 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:54:06.350687 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:54:06.350878 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:54:06.351475 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:54:06.351716 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:06.351781 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:54:06.351974  1168 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:06.520388 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:06.631574 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:54:06.634465 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:06.746124 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:06.762881 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:06.777766 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:06.865396 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 20:54:06.878221 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:06.918541 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:06.921277 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:07.146639 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:07.177870 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:07.191517 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:07.313910 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:07.324636 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 20:54:07.346489 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:07.360535 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:07.365257 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:07.498987 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:07.516688 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:07.547108 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:07.564365 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:07.571758 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:54:07.629345 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:07.686980 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:07.882496 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:07.917410 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:08.225792 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:08.231390 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:08.232121 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:08.446184 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:08.639122 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:08.943964 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:08.988864 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:09.023910 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:09.066578 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:09.096861 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:09.233845 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:09.263012 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:09.548211 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:09.606933 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:09.642871 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 20:54:09.724290 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:09.751695 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:09.782125 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:09.792902 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:09.811074 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:09.864879 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:54:09.884820 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:09.895637 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:10.006891 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:10.007040 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:10.024349 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:10.069818 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:10.136725 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:10.159034 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:10.218034 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:54:10.444499 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:10.525480 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:54:10.613121 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:10.646910 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:10.809725 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240502 20:54:10.845659  1168 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:10.845822  1168 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:54:10.846482  1168 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:54:10.847086 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:54:10.847301 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:54:10.848234 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:54:10.848323 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:10.848455 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:54:10.848621  1168 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:10.866310 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:10.972529 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:11.030483 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:11.116279 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20240502 20:54:11.391016 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:11.456820 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:11.489697 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:11.522835 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:11.577783 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:11.578476 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:54:11.580530  1063 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:11.580663  1063 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:54:11.581220  1063 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:54:11.581799 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:11.581976 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:54:11.582453 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:11.582577 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:11.582626 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:54:11.582777  1063 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:11.700778 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:54:11.866783 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:11.877362 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:54:11.926538 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:12.050329 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:12.075965 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:12.090446 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:12.122340 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 20:54:12.150720 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:12.267787 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:12.298979 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:12.358001 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:12.391078 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:12.440754 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:12.488142 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:12.522445 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:12.532601 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:12.585270 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:12.731941 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:12.783006 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:54:12.830089 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:13.101500 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:13.308315 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:13.324980 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:13.483552 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:13.558223 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:13.592183 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:13.615823 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:13.677713 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:13.894198 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:13.948643 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:14.073355 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:14.074759 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:14.179229 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:14.387359 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:14.452519 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:14.466107 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 20:54:14.642612 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:14.703258  1945 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:54:14.703480  1945 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:54:14.704557  1945 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:54:14.704946 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:14.705468 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:54:14.705857 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:14.705958 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:54:14.706111 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:14.706140  1945 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:54:14.725230 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:14.725404 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:14.747177 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:14.795856 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:14.824901 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:14.859758 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:14.883857 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:14.901705 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:14.950271 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:14.970741 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:15.136457 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:15.149857 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:15.378190 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:15.468091 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:54:15.534134 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 20:54:15.545836 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:15.682619 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:15.959651 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:16.002697 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:16.051275 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:16.075798 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:16.122107 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:16.150568 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:54:16.151708 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:16.193888 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:16.290199 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:16.667619 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:16.698760 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:16.727185 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:16.751751 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:54:16.801261 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 20:54:16.919832 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:54:16.921298 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:17.025987 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:17.077952 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:17.094090 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:17.172765 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:17.334481 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:17.390048 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:17.403735 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:17.409941 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:17.411481 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:17.451663 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:17.516863 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:17.603797 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:17.641047 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:17.679790 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:17.828836 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:54:17.873668 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:18.003198 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240502 20:54:18.059533  2247 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:54:18.059752  2247 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:54:18.060953 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:18.061349  2247 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
W20240502 20:54:18.061868 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:18.062355 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:54:18.062590 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:18.062666 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:54:18.063248  2247 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:54:18.101317 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:18.204720 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:18.250475 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240502 20:54:18.297046  2350 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:18.297362  2350 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:54:18.298120  2350 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:54:18.298472 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:54:18.298660 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:54:18.299723 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:18.300308 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:18.300407 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:54:18.300727  2350 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:18.510517 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:18.583787 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:18.596024 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:18.653043 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:18.705164 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:18.793491 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:18.796888 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:19.275928 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:54:19.292035  2247 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:54:19.292223  2247 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:54:19.292807  2247 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:54:19.294548 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:54:19.295313 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:54:19.295485 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:54:19.295703  2247 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:19.378667 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:19.396355 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:19.445411 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:19.485096 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 20:54:19.522791 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:19.575820 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:19.631901 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:19.831673 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:19.833047 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:19.854300 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:19.866114 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:19.875315 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:20.096328 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:20.116425 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:20.155181 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:20.206679 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:54:20.228026 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:20.299371 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:20.338085 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:20.412031 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:20.449355 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:20.568631 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 20:54:20.569343 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:20.729736 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:20.951985 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:20.955283 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:21.167685 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:21.215052 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:21.216933 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:21.239310 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240502 20:54:21.374704 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:21.540457 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:21.692610 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:21.764072 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 20:54:21.913347 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 20:54:21.937297 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:21.959136 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:22.009272 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 20:54:22.014894 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:22.048583 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:22.073010 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:22.149143 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:22.215977 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:22.258857 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:22.291347 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:22.348135 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:54:22.405229  2247 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:22.405405  2247 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
W20240502 20:54:22.406001 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 20:54:22.406078  2247 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:54:22.406672 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:22.406903 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:54:22.407508 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:22.407660 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:22.407721 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:54:22.407903  2247 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:54:22.493738  2247 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:54:22.493939  2247 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:54:22.494881  2247 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:54:22.495497 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:22.495723 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:54:22.496623 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:22.496753 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:54:22.497092  2247 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:54:22.497504 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:54:22.517613 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:22.551748 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:22.577394 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:22.638912 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:22.653731 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:54:22.662487 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:22.860818 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:22.908679 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:22.912262 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:23.114624 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:23.176092 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:23.466579 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:23.545871 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:23.663800 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:23.711776 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:23.713446 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:23.807308 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:23.873943 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:24.140504 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:24.246013 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 20:54:24.467414 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:24.467916 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:24.564009 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:24.588346 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:24.592371 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:24.615545 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:24.674248 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:54:24.697414  1168 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:24.697610  1168 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:54:24.698311  1168 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:54:24.699013 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:54:24.699245 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:54:24.700026 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:54:24.700270 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:24.700336 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:54:24.700536  1168 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:24.711375 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:24.719542 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:24.762379 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:24.768600 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:24.809355 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:24.836828 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:25.051793 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:54:25.173192 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 20:54:25.174417  2518 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:25.174585  2518 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:54:25.175191  2518 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:54:25.175853 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:54:25.176051 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:54:25.176656 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:25.176790 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:25.176846 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:54:25.177011  2518 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:25.322999 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:25.373926 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:25.379283 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:25.481204 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:25.505723 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:25.580551 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 20:54:25.714298 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:25.889568 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:26.097038 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:26.170001 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:26.181744 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:26.248345 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:26.283057 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:26.314924 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:54:26.371979 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:26.413013 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:26.537524 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 20:54:26.735682 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:26.855216 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:54:26.980181 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:26.995098 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:26.996179 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:27.011782 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:27.025144 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:27.103693 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240502 20:54:27.116400 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:27.182677 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:27.280465 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:27.309350 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:27.446259 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:27.491892 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:27.509495 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:27.521273 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:27.614456 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:27.616824 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:54:27.807235 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:27.892601 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:27.944869 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:27.982949 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:28.113049 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:28.190850 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:28.204214 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:28.432132 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:28.457491 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:28.504132 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:28.827620 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:28.863339 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:29.013825 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:29.042110 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:29.107781 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 20:54:29.153152 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:29.272009 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:29.272033 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:29.390906 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:29.432660 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:29.435676 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:29.487672 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:29.543804 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:29.583926 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240502 20:54:29.592104  3340 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:54:29.592293  3340 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:54:29.592826  3340 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:54:29.594439 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:54:29.594545 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:29.594655 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:54:29.594815  3340 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:29.925592 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:29.927067 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:29.945868 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:30.030135 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:30.103078 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:30.156033 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:30.184568 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:30.188877 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:54:30.340613 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:30.487669 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:30.587107 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:54:30.596733 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:30.602103 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:30.969550 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:30.982621 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:30.985428 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:31.097812 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:31.098214 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:31.150641 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:31.162267 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:31.217008 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:31.353394 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:31.360693 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:54:31.578717 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 20:54:31.771265 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:54:31.783987  3383 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:31.784219  3383 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:54:31.784904  3383 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:54:31.785145 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:54:31.785295 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:54:31.786170 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:54:31.786288 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:31.786381 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:54:31.786659  3383 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:31.824059 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:31.865435 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:31.921900 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:54:31.922760 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:31.926697 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:31.948701 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:31.951213 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:32.014737 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:32.098912 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:32.156005 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 20:54:32.205622 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:32.440099 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:32.463523 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:32.477623 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:32.499398 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:32.517946 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:54:32.670433 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:32.741235 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:32.920183 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:32.958956 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:33.165824 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:33.194684 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:33.315227 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:33.363217 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:33.542811 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:33.612206 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:33.656891 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:33.708469 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:33.871078 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:33.886921 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:33.982324 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:34.137771 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 20:54:34.235239 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:34.243927 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:34.330133 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:34.349002 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:34.356067 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:34.410598 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:34.455775 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:34.487046 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:34.648721 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:34.780441 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:34.802217 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:34.896127 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:34.922935 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:34.989252 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:35.016216 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:54:35.090400 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:35.194309 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:35.338124 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:35.524613 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:35.610143 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 20:54:35.613554 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:35.652158 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:35.761541 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 20:54:35.781641  3680 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:54:35.781821  3680 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:54:35.782539  3680 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:54:35.782933 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:35.783435 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:54:35.784087 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:35.784176 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:54:35.784344 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:35.784395  3680 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:54:35.846849 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:35.942366 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:36.201781 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:36.224645 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:36.340221 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:36.366397 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:36.393498 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:36.484859 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:36.629773 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:54:36.674785 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:36.725839 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 20:54:36.782548 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:36.868467 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:36.909324 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:36.924031 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:36.953537 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:37.019690 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:37.106379 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:37.124003 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:54:37.132767 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:37.147104 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:37.223906 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:37.233016 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:37.306316 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:37.317368 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:54:37.439081 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:37.671557 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:54:37.688822 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:37.716305 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 20:54:37.773211  4046 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:37.773478  4046 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:54:37.774310  4046 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:54:37.775146 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:54:37.775460 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:54:37.775950 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:37.776165 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:37.776247 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:54:37.776398  4046 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:37.845191 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:54:37.846091 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:37.853025 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:37.861641 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:37.894232 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:37.940124 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:38.185129 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:38.284384 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:38.417512 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:38.479322 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:38.736665 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:38.823556 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:38.862721 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:38.905516 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:38.911948 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:38.996706 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:39.057225 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:39.125918 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:39.188946 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 20:54:39.257520 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:39.265964 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:39.429334 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:39.517190 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240502 20:54:39.555780  4228 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:39.555979  4228 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:54:39.557157  4228 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:54:39.558035 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:39.558316 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:54:39.559310 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:39.559463 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:39.559525 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:54:39.559762  4228 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:39.625174 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:39.704089 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:39.706211 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:39.734848 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:39.741606 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:39.754033 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:39.841311 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:39.921461 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:40.074589 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fdd2311ec0b645f5a79098a32c6deba2: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:54:40.210736 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:40.256919 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:40.285281 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:40.308557 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:40.371886 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:40.381134 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:54:40.383292 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:40.479954 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:40.613989 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:40.713347 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 20:54:41.037004 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:41.135843 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:41.201637  4293 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:41.201859  4293 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:54:41.202497  4293 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:54:41.202847 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:41.203050 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:54:41.204097 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:41.204229 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:41.204313 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:54:41.204510  4293 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:41.239794 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:41.352384 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:41.393522 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:41.426503 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:41.464883 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:41.486191 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:41.520408 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:41.521093 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:41.561609 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 20:54:41.598030 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:41.660373 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:41.803805 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:54:41.881008  3655 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:41.881230  3655 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:54:41.881886  3655 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
W20240502 20:54:41.882481 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 20:54:41.882455 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:54:41.882642 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:54:41.883229 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:54:41.883376 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:41.883482 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:54:41.883651  3655 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:41.907874 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:42.118534 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:42.229425 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:54:42.230363 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:42.247627 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:42.250375 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:42.341174 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:42.384446 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 20:54:42.414683 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:42.461869 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:42.573622 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:42.628835 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:42.632318 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:42.771998 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:42.892982 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:42.893800 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:54:42.935639 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:42.988740 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 20:54:43.033295  4293 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:54:43.033526  4293 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:54:43.034250  4293 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:54:43.034873 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:43.035120 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:54:43.035998 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:43.036074 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:54:43.036382  4293 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:54:43.036482 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:54:43.211421 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:43.223135 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:43.438074 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:43.554688 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:43.689961 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 20:54:43.717732  3655 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:43.717974  3655 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:54:43.719015  3655 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:54:43.719867 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:54:43.720283 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:54:43.720664 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:54:43.721274 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:43.721364 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:54:43.721565  3655 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:43.733853 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:43.773182 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:43.819692 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:43.923516 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:44.053184 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:44.115463 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:44.170161 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:44.211346 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 20:54:44.245945 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:44.350400 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:44.398963 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:44.448992 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:44.574010 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:44.636193 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 20:54:44.708675  4293 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:54:44.708853  4293 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:54:44.709537  4293 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:54:44.710094 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:44.710562 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:54:44.711272 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:44.711370 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:54:44.711587 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:44.711658  4293 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:54:44.714293 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:44.744405 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240502 20:54:44.917131  4760 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:44.917447  4760 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
W20240502 20:54:44.917824 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240502 20:54:44.918140  4760 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:54:44.918674 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:54:44.918877 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:54:44.919319 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:44.919840 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:44.920390 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:44.920475 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:54:44.920699  4760 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:45.013021 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:45.114665 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:45.271623 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:45.280782 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:45.294957 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:45.332378 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:45.549950 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:45.549950 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:54:45.590056 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:45.787048 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 20:54:45.848205 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:45.902762 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:45.974725 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:46.093904 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:46.245771 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:46.250808 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:46.315526 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:46.385282 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:46.558630 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:46.611552 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:46.636216 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 20:54:46.704380 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:46.759896 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:46.806455 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:46.911008 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:46.932032 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:46.952409 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240502 20:54:46.971782 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:46.996336 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:47.021190 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:47.258479 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:54:47.259541 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:47.373658 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:47.386449 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 20:54:47.460376 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:47.461109 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:47.588850 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:47.636576 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:47.815431 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:47.826867 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:47.899168 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:47.934767 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:48.033860 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:54:48.034922 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:48.274921 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:48.342689 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:48.573722 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:48.646414 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:48.669482 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:48.754472 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:48.881994 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:49.059617 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:49.122217 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:49.127748 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:49.162798 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:49.194943 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:49.199275 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 20:54:49.348826 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:49.361838 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 20:54:49.363262  5073 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:54:49.363421  5073 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:54:49.363921  5073 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:54:49.365510 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:54:49.365670 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:49.365731 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:54:49.365921  5073 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:49.366714 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:49.398264 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:49.465498 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:49.480031 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:49.745072 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:49.910849 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:49.999269 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:50.057710 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:50.112749 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:50.139204 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:50.150175 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:50.166007 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:50.231284 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:50.380811 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:50.496841 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:50.619483 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:54:50.721215 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:50.837774 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:54:50.938050 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:51.033351 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:51.062336 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:51.070791 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:51.283002 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:51.394151 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:51.399663 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:51.512161 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:51.529168 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:51.630556 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 20:54:51.758093 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:51.768786 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:51.865708 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:51.884886 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:51.933029 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:51.955343 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:51.985399 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:52.040937 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 20:54:52.091838 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:52.355692 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 20:54:52.356707 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:52.449950 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 20:54:52.450870 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:52.521978 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:52.576848 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:52.603866 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:52.635437 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:52.699287 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:52.727667 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:52.817977 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:52.937898 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:52.951637 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:53.280690 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:54:53.316681 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:53.368686 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:53.400597 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:53.662561 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:53.719138 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:53.772796 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:53.886699 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:53.924552 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 20:54:53.948362 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:53.977217 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:53.992177 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:54.037194 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:54.103746 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:54.255805 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:54.403236 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:54.494079 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:54.505849 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:54.554234 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:54.729702 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:54.748705 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:54.888715 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:55.085628 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:55.104873 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:55.208045 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:55.250414 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:55.309170 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:55.369784 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:55.383886 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:55.678678 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:55.707296 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:54:55.831408 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:55.845472 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 20:54:55.861186 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:54:55.875259 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:55.950161 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:55.990067 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:56.176705 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:56.289849 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:54:56.304941 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 20:54:56.319979 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:56.352247 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:56.364369 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:56.506690 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:56.572094 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:56.591367 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:56.641564 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:56.696388 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:56.811216 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:56.846698 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:57.000900 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:57.027943 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:57.130884 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:54:57.195314 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:57.226907 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:57.432654 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:54:57.450855 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:54:57.451699 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:57.703692 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:57.722648 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:54:57.748585 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:57.809420 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:54:57.939147 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:57.946477 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:54:58.050961 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:54:58.333101 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:58.344115 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
I20240502 20:54:58.351766  5897 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:54:58.351964  5897 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:54:58.352717  5897 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:54:58.353600 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:54:58.353860 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:54:58.355240 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:54:58.355379 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:58.355414 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:54:58.355698  5897 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:54:58.497324 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:58.498176 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:54:58.536564 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:58.543170 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:54:58.774027 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:54:58.821367 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:54:58.827425 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:54:58.849699 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:58.936458 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 20:54:58.950526 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:54:59.053547 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:54:59.155201 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:54:59.204895 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:59.247012 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:54:59.265427 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:54:59.300170 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 20:54:59.344947  5484 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:54:59.345194  5484 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:54:59.346159  5484 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:54:59.346696 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:54:59.347057 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:54:59.347748 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:54:59.347890 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:54:59.348145 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:54:59.348165  5484 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:54:59.353790 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:54:59.431883 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:54:59.505342 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:54:59.561966 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:54:59.867831 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:54:59.921708 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:00.109473 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:00.288307 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:00.349576 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 20:55:00.390792 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 20:55:00.419533 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:00.477916 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:00.667754 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:00.764415 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:00.855598 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 20:55:00.860642 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:55:00.904909 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:55:00.929993 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:01.089341 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:01.183101 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:01.208667 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:01.222587 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 20:55:01.398597 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 20:55:01.480104 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 20:55:01.520927 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:01.532711 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:01.592501 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:01.630914 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:01.681795 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:01.744166 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:01.848212 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 20:55:01.913920 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
I20240502 20:55:01.929517  6239 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:01.929714  6239 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:55:01.930382  6239 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:55:01.930929 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:55:01.931152 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:55:01.931823 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:01.931977 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:01.932029 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:55:01.932216  6239 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:01.935204 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 20:55:01.948038 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:55:02.040457  6239 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:02.040668  6239 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:55:02.041343  6239 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:55:02.041957 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:55:02.042208 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:55:02.042734 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:55:02.042933 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:02.043000 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:55:02.043287  6239 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:02.069362 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 20:55:02.200378 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 20:55:02.322958 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:02.342702 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:55:02.451856 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240502 20:55:02.465698 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:02.555521 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 20:55:02.678431 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:02.751346 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:02.884712 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:02.912915 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:03.092263 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:03.096668 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:03.108470 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:03.278662 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:03.334758 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:55:03.377913 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:03.426069 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:03.566308 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:03.584977 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:03.608848 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:03.729110 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 20:55:03.963837 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:04.007799 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:04.120431 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:04.147935 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:04.153113 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:04.261462 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:04.266660 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 20:55:04.390333 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:04.408533 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:04.437171 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:04.457778 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:04.512121 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:04.530306 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:04.567562 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:04.655297 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:55:04.861667  6387 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:04.861869  6387 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:55:04.862540  6387 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:55:04.863355 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:55:04.863554 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:55:04.863956 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:04.864143 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:04.864207 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:55:04.864440  6387 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:04.869802 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:04.988816 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:05.171125 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:05.224189 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:55:05.271138  6494 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:55:05.271333  6494 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:55:05.272105  6494 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:55:05.272601 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:55:05.273511 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:55:05.274230 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:55:05.274315 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:55:05.274533 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:05.274542  6494 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:55:05.428246 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:05.532310 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:05.673013 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:05.676941 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:05.711180 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:05.751258 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:05.877796 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 20:55:05.894960 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:05.970219 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 20:55:06.051613 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:06.086053 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:06.136058 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:06.149682 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:06.612735 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:06.613833 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:06.689402 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:06.700131 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:06.723456 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:06.823542 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:06.826347 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:06.832450 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:06.847019 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 20:55:06.977378 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:06.980391 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:07.042655 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:07.043195 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:07.050209 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
I20240502 20:55:07.061193  6494 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:55:07.061403  6494 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:55:07.061981  6494 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:55:07.063447 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:55:07.066022 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:07.066169 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:55:07.066371  6494 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:07.185771 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:07.204335 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:07.249277 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:07.495150 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:55:07.520278 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:07.590258 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 20:55:07.609241 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:55:07.695461 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 20:55:07.914636 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:07.968914 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:08.061350 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:08.205191 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:08.240176 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:08.459205 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:08.474547 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:08.518515 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:08.568998 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 20:55:08.574262  6494 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:55:08.574469  6494 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:55:08.575165  6494 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:55:08.576542 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:55:08.576750 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:55:08.576844 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
W20240502 20:55:08.576956 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 20:55:08.577147  6494 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:08.627156 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:08.643703 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:08.990564 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:09.127321 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:09.178205 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:09.221594 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:09.235658 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 20:55:09.351994 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:09.377377 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:09.399916 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:09.439785 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:09.506429 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:09.513796 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:09.533993 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:09.548521 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:09.549422 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:09.632596 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:09.807940 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:09.844395 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:09.906777 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:55:09.934758 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:10.093668 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 20:55:10.145017  6980 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:10.145231  6980 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:55:10.145831  6980 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:55:10.146287 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:55:10.146457 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:55:10.147505 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:10.147667 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:10.147732 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:55:10.147960  6980 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:10.286361 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:10.342530 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:10.601183 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:10.740587 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:10.917999 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 20:55:10.918989 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:10.994755 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 20:55:11.013366 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:11.020318 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:11.026098 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:11.035338 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:11.131556 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:11.225208 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:11.556959 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:11.635227 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:11.684319 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:11.687084 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 20:55:11.722882 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:11.744571 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:11.821223 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:11.842578 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:11.855876 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:11.929347 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:11.986308 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:12.129333 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:12.155508 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:12.201591 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:12.235369 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:12.278108 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:12.542286 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:55:12.543458 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:12.587495 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:12.629434 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 20:55:12.660449 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:55:12.754312 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:12.861974 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:13.015707 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:13.277792 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:13.378257 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:13.419279 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:13.420912 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:13.463130 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:13.502724 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:13.536343 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 20:55:13.568933 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:13.617710 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:13.682622 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:13.938565 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:13.983209 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:14.060559 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:14.136025 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:14.170718 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:14.259894 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:14.286865 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:14.287690 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 20:55:14.428869 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:14.481176 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:14.490965 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:14.597440 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:14.648142 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:14.652629 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:14.655696 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:14.873993 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:14.901261 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:15.074100 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:15.301143 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:15.438470 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
I20240502 20:55:15.590025  7402 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:55:15.590224  7402 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:55:15.591001  7402 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:55:15.591492 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:55:15.591679 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:55:15.592206 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:15.592290 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:55:15.592464  7402 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:55:15.592756 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:55:15.617980 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:15.727002 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:15.809159 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:15.871999 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:15.939153 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 20:55:16.012751 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:16.065908 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:16.103441 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:16.110944 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 20:55:16.116925 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:16.185433 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:16.234279 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:16.499619 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:16.553480 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:16.558569 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:16.665234 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:16.686831 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:16.854013 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:16.866111 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:16.872141 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:16.883240 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 20:55:16.903254 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:16.920451 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:16.921170 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:16.954382 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:16.983633 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:17.193445 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:17.544888 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:17.545018 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 20:55:17.631098 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:17.656507 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:55:17.714836 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:17.806999 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:17.909096 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:55:17.919188 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:18.220935 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:18.237283 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:18.368662 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:18.476682 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:18.500501 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:18.520244 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:18.550565 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:18.570653 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:18.618291 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 20:55:18.768138 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:18.958420 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 20:55:19.016247 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:19.044183 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:19.079829 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:19.133666 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:19.169806 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:19.237540 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:19.281512 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:19.331568 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:19.373387 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:19.379874 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:19.408334 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:19.438838 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 20:55:19.444430 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:19.459386 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240502 20:55:19.486481  7801 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:19.486686  7801 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:55:19.487573  7801 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:55:19.488150 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:55:19.488354 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:55:19.488785 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:55:19.488878 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:19.488967 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:55:19.489168  7801 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:19.681524 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:19.830467 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:20.055853 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:20.074535 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 20:55:20.355293 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:20.525801 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:20.551610 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:20.688943 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:20.691208 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 20:55:20.748145 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:20.860067 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
I20240502 20:55:20.880466  7843 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:20.880676  7843 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:55:20.881506  7843 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:55:20.882247 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:55:20.882524 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:55:20.883327 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:20.883477 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:20.883544 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:55:20.883761  7843 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:20.898965 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:21.008118 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 20:55:21.034507 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:21.068465 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 20:55:21.277261 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:21.313881 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:21.385013 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:21.485433 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:21.490921 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:21.501341 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:21.521461 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:21.639804 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:21.667479 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:21.716248 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:21.749634 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 20:55:21.760237 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:21.910837 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 20:55:21.998675 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 20:55:22.011740 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 20:55:22.039148 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:22.069337 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 20:55:22.307488 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
I20240502 20:55:22.339497  7801 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:22.339668  7801 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:55:22.340329  7801 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:55:22.340808 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:55:22.340972 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:55:22.341467 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:55:22.341836 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:55:22.341921 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:55:22.342124  7801 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:22.568013 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:55:22.604171 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 20:55:22.704325 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:55:22.721288 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:22.839164 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:22.924701 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:22.931787 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240502 20:55:22.934615 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:23.023836 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:23.074236 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:23.076745 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 20:55:23.333040 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 20:55:23.373540 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:23.373548 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:23.379845 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:23.400722 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:23.579978 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:23.756878 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:23.799943 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:23.900537 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:23.919889 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:23.998505 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:24.028292 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:24.094638 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:24.119515 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:24.134086 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:24.221747 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:24.273540 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:24.309942 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:24.509435 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 20:55:24.532150 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:24.552263 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:24.664428 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:24.822826 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:25.044091 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:25.140515 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:25.207671 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:25.437367 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:25.533524 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:25.571935 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 20:55:25.619647 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:25.652128 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:25.662001 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:25.736472 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:25.785979 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 20:55:25.805233  8258 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:55:25.805408  8258 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:55:25.806028  8258 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:55:25.806335 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:55:25.806557 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:55:25.807359 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:55:25.807435 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:55:25.807503 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:25.807577  8258 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:55:25.872016 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:25.887706 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:25.990178 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:26.052016 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 20:55:26.061479 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:26.086869 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:26.210203 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:26.434702 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:26.514577 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:26.563162 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:26.565639 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:26.588222 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:26.650753 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:26.748159 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:26.778256 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:26.778471 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:26.962662 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:26.967577 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:26.970355 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:27.304869 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 20:55:27.409794 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:27.511816 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:27.567055 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:27.720789 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:55:27.727880 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:27.808527 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240502 20:55:27.813853 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:27.816535 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:28.164917 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:28.200006 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:28.221725 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:28.257097 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:28.268893 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 20:55:28.273234 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240502 20:55:28.300438  8258 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:55:28.300603  8258 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:55:28.301553  8258 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:55:28.303030 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:55:28.303213 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:28.303308 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:55:28.303481  8258 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:28.308677 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:28.313223 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:28.389770 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:28.405900 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:28.503151 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:28.635417 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:28.697386 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:28.731227 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:28.835183 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:28.883795 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:28.890681 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:29.134728 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:29.180171 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:29.249884 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:29.312683 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:29.367985 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:29.393460 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:29.510095 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:29.784193 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:29.785319 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:29.926203 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 20:55:29.973378 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:30.002024 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:30.202374 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:30.459321 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:30.490411 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:30.521195 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:30.668823 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:30.712406 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:30.795473 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:30.803166 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 20:55:30.890328 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:30.946326 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:30.946899 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:30.975222 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:30.996346 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:31.038333 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:31.078352 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:31.086221 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:31.127679 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 20:55:31.157521 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:31.447851 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:31.464521 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:31.631493 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:31.742525 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:31.754686 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:31.882961 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:31.913270 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:31.953893 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240502 20:55:32.008010  8930 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:32.008285  8930 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:55:32.008987  8930 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:55:32.009764 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:55:32.010001 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:55:32.010655 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:55:32.011297 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:55:32.011369 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:55:32.011575  8930 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:32.043728 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:32.092324 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:32.342072 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:32.385959 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 20:55:32.533476 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:32.704059 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:32.739168 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:55:32.739946 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:32.771878 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:32.821321 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:32.857460 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:55:32.971678 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 20:55:32.993108  8653 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:32.993264  8653 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:55:32.993865  8653 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:55:32.994349 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:55:32.994509 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:55:32.995680 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:32.995781 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:32.995849 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:32.995926 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:55:32.996132  8653 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:33.019492 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:33.023481 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:33.119782 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 20:55:33.282738 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 20:55:33.292867 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:33.372830 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:33.372946 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:33.475682 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 20:55:33.503304  9070 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:55:33.503559  9070 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:55:33.504257  9070 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:55:33.505548 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:55:33.505759 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:55:33.505882 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:55:33.506059  9070 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:33.514972 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:33.590711 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:33.668401 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:33.767617 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:33.822870 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:33.961691 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:34.075127 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:34.114149 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:34.195255 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:34.305930 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:34.413118 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:34.435631 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:34.470737 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:34.606122 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:34.666916 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:34.756017 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:34.805136 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:34.849205 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 20:55:35.010820 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:35.153517 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:35.272110 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:35.304625 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:35.488776 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:35.510511 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:35.570852 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:35.647132 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:35.656638 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:35.719053 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 20:55:35.746879 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:36.006363 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:36.041850 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:36.083194 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:36.153147 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 20:55:36.154659 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:36.187426 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:36.189417 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 20:55:36.271749  9289 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:36.271941  9289 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:55:36.272640  9289 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:55:36.273277 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:55:36.273494 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:55:36.274161 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:36.274670 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:36.274962 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:55:36.275154  9289 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:36.484129 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:36.595877 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:36.676556 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:36.720137 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:36.813978 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:36.877004 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:36.903681 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:36.913925 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:36.974756 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:36.998899 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:37.049835 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:37.064589 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:37.251273 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:37.490435 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 20:55:37.536952 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:37.738279 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:37.758371 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:37.855130 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:37.871403 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:55:37.881867 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:55:37.883106 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:37.899231 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:37.951371 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:37.958726 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 20:55:37.959780 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:38.070863 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:38.093919 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:38.242755 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:38.475342 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:38.493350 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:55:38.513303 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:38.526571 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 20:55:38.579128 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:38.662860 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:38.712080 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:38.728310 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:38.845445 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:39.016781 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:39.044438 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:39.070899 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:39.099604 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:39.194130 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:39.385928 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:39.401651 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:39.467222 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:39.491587 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:39.525751 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:39.666658 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:39.699347 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:39.717860 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:39.920462 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:39.932503 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 20:55:39.991003 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:40.125293 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fdd2311ec0b645f5a79098a32c6deba2: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:40.220053  9688 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:40.220301  9688 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:55:40.221014  9688 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:55:40.221524 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:55:40.221716 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:55:40.223454 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:40.223632 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:40.223695 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:55:40.223933  9688 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:40.263026 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:40.276079 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:40.308598 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:40.333842 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 20:55:40.421947  9070 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:40.422188  9070 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:55:40.422910  9070 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:55:40.423903 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:55:40.424124 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:55:40.426224 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:55:40.426708 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:40.426803 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:55:40.427091  9070 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:40.451439 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:40.561844 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:40.598495 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:40.777376 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 20:55:40.822319 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:40.829706 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:41.002455 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:41.070345 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:41.148932 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:55:41.196583 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:41.206099 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 20:55:41.213639 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:41.245702 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:41.274133 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:41.354274 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:41.566345 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:41.743587 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:41.783877 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:41.799101 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:41.843492 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:55:41.889863  9775 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:41.890156  9775 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:55:41.890956  9775 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:55:41.891541 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:55:41.891767 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:55:41.893147 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:41.893304 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:41.893347 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:55:41.893983  9775 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:41.911024 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:42.060729 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:42.062952 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:42.171226 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
I20240502 20:55:42.239790  9775 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:55:42.240082  9775 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:55:42.240767  9775 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:55:42.242110 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:55:42.242383 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:55:42.242498 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:55:42.242698  9775 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:42.320364 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:42.436908 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:42.480743 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:42.513726 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:42.556265 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 20:55:42.565192 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:42.684772 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:42.743352 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:42.854485 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:42.990689 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:55:43.043397 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240502 20:55:43.044718 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:43.273979 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:43.291460 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:43.312605 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:43.379125 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 20:55:43.512599 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:43.548326 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:43.673683 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:43.691573 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:43.757047 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:55:43.757892 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:43.815097 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:43.902734 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:44.043380 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:44.073896 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:55:44.212373  9775 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:55:44.212575  9775 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:55:44.213385  9775 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:55:44.214186 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:55:44.214757 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:55:44.215477 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:44.215600 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:55:44.215670 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:55:44.215792  9775 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:55:44.221145 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:44.397656 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:44.419148 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:44.422071 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 20:55:44.440304 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:44.458204 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:44.589666 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:44.749990 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:44.837750 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:44.870395 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:44.934360 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 20:55:45.002317 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:45.185355 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 20:55:45.244868 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:45.342480 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 20:55:45.350574 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:45.359566 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 20:55:45.518263 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:45.578462 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:45.731138 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:45.742413 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 20:55:45.980428 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 20:55:46.016654 10177 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:55:46.016822 10177 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:55:46.017413 10177 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:55:46.018661 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:55:46.018797 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:46.018879 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:55:46.019106 10177 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:46.037468 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:46.204003 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:46.212888 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:46.225914 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:46.245601 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 20:55:46.313210 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:46.316071 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:46.457999 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:46.623889 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 20:55:46.658460 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:46.691169 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 20:55:46.695576 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 20:55:46.863842 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:46.879395 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:47.006343 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 20:55:47.140298 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:47.254662 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:47.403457 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:47.446344 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:47.550190 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:47.606022 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:47.658196 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240502 20:55:47.690057 10285 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:55:47.690382 10285 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:55:47.690963 10285 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:55:47.692098 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:55:47.692214 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:47.692283 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:55:47.692449 10285 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:47.728905 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:47.784929 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:47.795974 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 20:55:47.886945 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:48.040045 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240502 20:55:48.065629  9289 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:48.065810  9289 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:55:48.066668  9289 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:55:48.067159 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:55:48.067346 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:55:48.067993 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:48.068142 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:48.068187 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:55:48.068475 10410 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:48.086688 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240502 20:55:48.087728 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:48.087976 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:55:48.099596 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:48.167392 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:48.242991 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:48.339754 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:48.648357 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:48.658682 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 20:55:48.714246 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 20:55:48.737226 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:48.804711 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:48.857764 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:48.900200 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:55:48.901372 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:48.962833 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240502 20:55:49.126907 10285 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:49.127087 10285 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:55:49.127594 10285 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:55:49.128451 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:55:49.128646 26959 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:55:49.129266 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:55:49.129590 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:49.129660 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:55:49.129854 10285 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:49.186297 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:49.222750 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:49.265873 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:49.421456 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:49.423169 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:49.429435 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:49.678700 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:49.760078 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:49.897156 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:50.124634 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:50.126238 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:50.169375 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:50.239591 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:50.294021 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:50.363384 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 20:55:50.378377 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:50.436782 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:50.439851 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:50.679410 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:50.683938 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:50.740695 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:50.832468 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:50.889510 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:50.989588 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:51.024572 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 20:55:51.267203 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:51.275246 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 20:55:51.293386 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:51.296718 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:51.359228 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:51.590131 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:51.604787 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:51.634069 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:51.666887 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:51.732087 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:51.780349 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:51.985137 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:52.023866 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:52.033232 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:52.183157 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:52.216683 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 20:55:52.295327 10748 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:52.295516 10748 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:55:52.296161 10748 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:55:52.296573 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:55:52.296747 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:55:52.297305 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:55:52.297554 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:55:52.297616 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:55:52.297773 10748 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:52.695283 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:52.714439 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:52.752359 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 20:55:52.753146 10770 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:52.753269 10770 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:55:52.753814 10770 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:55:52.754529 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:55:52.754690 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:55:52.757546 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:55:52.757828 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:52.757926 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:55:52.758148 10770 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:52.796897 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:52.806370 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:52.840848 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:52.885941 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 20:55:52.910872 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:53.001977 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:53.106916 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:55:53.125816 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:55:53.126737 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:53.358618 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:53.359850 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:53.425566 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:53.431996 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:53.552558 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:53.580384 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 20:55:53.587165 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:55:53.737824 10878 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:55:53.738210 10878 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:55:53.739254 10878 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:55:53.740254 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:55:53.740779 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:55:53.741238 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:55:53.741361 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:53.741576 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:55:53.741834 10878 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:55:53.766424 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:53.859467 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:53.896029 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:54.054630 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:54.072870 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:54.087868 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 20:55:54.150962 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:54.169674 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:54.214613 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:54.282097 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:54.373656 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:54.466264 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:54.491953 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 20:55:54.560643 10748 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:54.560891 10748 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:55:54.561657 10748 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:55:54.562130 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:55:54.562319 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:55:54.563292 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:55:54.563498 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:54.563645 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:55:54.564077 10748 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:54.823177 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:54.880044 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:54.924190 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20240502 20:55:55.108857 10878 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:55:55.109110 10878 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:55:55.109771 10878 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:55:55.111183 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:55:55.111625 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:55.111718 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:55:55.111896 10878 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:55.176323 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:55.225358 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:55.232306 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:55.336907 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:55.409833 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:55.506152 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:55.519011 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:55.564059 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 20:55:55.690862 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:55.751667 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:55.791349 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:55.862582 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:55.936823 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 20:55:55.949201 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:55:55.979923 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:56.127118 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:56.187673 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:56.220393 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:56.234586 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:56.259686 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:56.303507 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 20:55:56.564805 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:56.670915 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:56.735802 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:56.739362 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:56.956723 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:56.964885 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:56.974539 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:55:57.021170 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:57.083285 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:57.337529 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:57.514056 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:55:57.524855 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:57.668051 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:55:57.678354 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:57.785796 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:55:57.866387 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:55:57.974738 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 20:55:58.006999 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:58.109828 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:55:58.118106 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:55:58.127604 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:58.168216 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 20:55:58.169032 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 20:55:58.187657 10878 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:55:58.187855 10878 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:55:58.188545 10878 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:55:58.188803 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:55:58.189007 26959 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:55:58.190281 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:55:58.190450 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:55:58.190569 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:55:58.190744 10878 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:55:58.215862 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:58.228106 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:55:58.350504 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:55:58.400053 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:58.531772 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 20:55:58.549412 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:55:58.563272 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:55:58.623602 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:55:58.975755 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:55:59.034304 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:55:59.156752 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 20:55:59.247241 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:59.324833 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:55:59.402135 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:55:59.402248 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:55:59.444211 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:55:59.561337 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:59.610781 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:55:59.687427 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:55:59.785542 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:55:59.913302 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:00.074610 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:00.242349 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:00.248162 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:56:00.312287 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:56:00.336887 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:00.344445 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:00.422453 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:56:00.504989 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:00.533638 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:56:00.551810 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 20:56:00.727290 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:00.790077 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:00.850378 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:00.913579 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:00.976271 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:56:01.001284 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:56:01.023591 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:56:01.315450 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 20:56:01.339089 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:01.343858 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 20:56:01.366741 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:01.698477 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:01.720700 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:01.754570 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:01.823535 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:01.825949 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:56:01.960202 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:02.113808 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:02.239292 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:02.273495 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:56:02.359113 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:56:02.470921 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:02.673408 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:56:02.676658 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:02.683424 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:02.735220 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:02.804173 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 20:56:02.824074 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:02.864001 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:02.913312 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:02.957914 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:56:02.973636 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:03.067708 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:03.140772 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 20:56:03.185328 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:56:03.385457 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:03.459890 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:03.537487 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:03.640580 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:03.758882 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:03.776379 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:03.884904 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:03.914615 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 20:56:04.042990 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:04.096215 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:04.101197 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:04.144166 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:04.186815 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:04.269958 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:04.370800 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:56:04.610841 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:04.677520 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
I20240502 20:56:04.778447 11568 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:04.778614 11568 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:56:04.779197 11568 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:56:04.780398 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:56:04.780587 26959 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:56:04.781054 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:56:04.782193 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:56:04.782325 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:56:04.782546 11568 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:04.808609 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:04.860348 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:04.972443 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:05.010412 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:05.146682 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:05.222751 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:05.254190 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:05.270988 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:05.351269 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 20:56:05.352329 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 20:56:05.369331 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 20:56:05.404330 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:05.447903 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:05.507997 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:05.671638 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 20:56:05.676949 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
I20240502 20:56:05.931694 11568 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:05.931860 11568 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:56:05.932900 11568 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:56:05.933390 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:56:05.933614 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:56:05.934463 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:05.934608 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:05.934665 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:56:05.934819 11568 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:06.005846 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:06.133229 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:06.151928 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:06.177980 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:06.214071 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:06.331969 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 20:56:06.374995 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:06.376091 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 20:56:06.419340 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:06.480814 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:06.493209 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 20:56:06.520841 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:06.617625 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 20:56:06.754689 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:06.810467 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 20:56:07.330058 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:07.419684 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:07.515357 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:07.519400 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 20:56:07.557050 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:07.603533 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:07.709540 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:07.738662 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:07.829528 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:07.831049 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:07.848084 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 20:56:07.882779 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:07.988422 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:08.078349 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:08.079401 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:08.247349 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:08.268000 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:56:08.268954 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:08.288024 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 20:56:08.348057 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:08.537355 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:08.690943 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 20:56:08.695036 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:08.755267 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:08.774436 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 20:56:08.841323 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 20:56:08.855896 11970 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:56:08.856114 11970 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:56:08.856688 11970 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:56:08.858314 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:56:08.858414 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:08.858477 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:56:08.858616 11970 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:08.891078 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
I20240502 20:56:08.891937 12241 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:56:08.892082 12241 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:56:08.892793 12241 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:56:08.893455 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:56:08.893519 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:56:08.894436 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:08.894521 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:56:08.894629 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:56:08.894711 12241 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:56:08.905738 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:08.924327 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:08.997033 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:09.262271 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:09.320322 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:09.352536 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:09.446370 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:56:09.748301 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:09.846534 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:09.878397 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:09.910740 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:56:09.912779 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:09.934006 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:09.971213 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:10.083046 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 20:56:10.217372 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:10.260573 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:10.271934 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:10.359513 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:10.452333 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:10.466384 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:10.610723 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:10.669778 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:10.702041 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:10.727809 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:10.810680 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:10.846767 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:11.031940 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:11.085489 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:11.154243 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:11.172087 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:11.272406 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 20:56:11.338021 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 20:56:11.487674 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:11.500818 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:11.578279 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:11.663903 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:11.707271 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:11.939671 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:11.987218 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:12.205051 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:12.206562 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:12.290370 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:12.291097 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:12.305464 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:12.466181 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:12.654016 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:12.702765 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 20:56:12.741099 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:12.742635 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:12.764420 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:12.835753 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:12.867094 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:13.054450 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:13.115298 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:13.235531 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:13.248795 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:13.305666 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 20:56:13.328917 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:13.366868 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:13.408079 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:13.455646 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:13.489405 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240502 20:56:13.609325 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:13.753554 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 20:56:13.867743 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:13.898723 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:14.150514 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:14.178958 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:14.208874 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:14.275002 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:14.282511 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:14.448381 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:14.505304 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:14.565790 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:56:14.575433 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:14.640822 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:14.765383 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:14.902860 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:14.941634 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:14.943248 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:15.014838 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:15.140448 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:15.194612 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:15.266722 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:15.286180 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 20:56:15.408706 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:15.482543 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:15.625841 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:15.668387 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:15.674842 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:15.768765 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:15.831656 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:15.940855 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:16.009251 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:16.217615 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:16.326189 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:16.353263 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 20:56:16.368450 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 20:56:16.462812 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:16.573164 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:16.612577 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:16.627380 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:16.709213 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:16.889336 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:16.966771 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:16.966771 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20240502 20:56:16.971482 12431 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:56:16.971773 12431 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:56:16.972529 12431 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:56:16.973130 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:56:16.973307 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:56:16.973825 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:56:16.973968 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:56:16.974157 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:16.974175 12431 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:56:17.097363 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:17.233448 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:17.264051 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:17.296650 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:17.335425 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:17.467409 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:17.529592 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:17.619318 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:17.681929 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:17.823110 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 20:56:17.904935 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:17.908546 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:17.991004 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:18.063774 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
I20240502 20:56:18.129923 12431 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:18.130107 12431 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:56:18.130661 12431 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:56:18.131148 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:56:18.131330 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:56:18.132179 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:56:18.132292 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:18.132366 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:56:18.132526 12431 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:18.189584 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:18.219091 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:18.230348 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:18.396575 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:18.440282 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 76 similar messages]
I20240502 20:56:18.490705 12431 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:56:18.490909 12431 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:56:18.491425 12431 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:56:18.492647 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:56:18.492740 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:18.492842 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:56:18.493158 12431 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:18.531109 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:56:18.577258 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:18.580603 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:18.781011 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:18.823177 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 20:56:18.832576 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:18.874534 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 20:56:18.888469 12955 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:18.888682 12955 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:56:18.889726 12955 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:56:18.890185 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:56:18.890367 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:56:18.891269 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:18.891628 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:18.891706 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:56:18.891875 12955 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:19.065898 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:19.107669 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:19.130659 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:19.300721 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:19.411810 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:19.417332 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:19.490103 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:19.615520 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:19.642129 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:19.648340 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:19.705011 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:56:19.727264 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:19.744334 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:19.827932 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:20.008747 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:20.098757 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:20.225575 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 20:56:20.285995 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:20.288547 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:20.358173 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:20.393803 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:20.591595 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:20.729946 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:20.848739 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:20.851632 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:20.964744 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:21.038177 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:21.049368 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:21.116417 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:21.368710 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:21.399240 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 20:56:21.458636 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:21.533262 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 20:56:21.582276 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:21.583328 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:21.632277 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:21.738832 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:21.818627 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:21.844256 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:22.005527 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:22.060017 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:22.081359 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:22.111191 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:56:22.129158 13445 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:22.129384 13445 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:56:22.130071 13445 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:56:22.130407 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:56:22.130580 27359 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:56:22.131759 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:22.131855 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:22.131903 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:56:22.132079 13445 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:22.219467 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:22.301760 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:22.383970 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:22.504086 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:22.504413 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:22.710201 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 20:56:22.727262 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:22.781908 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:22.873854 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:22.873854 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:23.035874 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:23.118945 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:23.420169 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:23.448639 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 20:56:23.498533 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:23.607965 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 20:56:23.633639 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:23.673347 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:23.713312 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:23.760702 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 20:56:23.858542 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:23.872373 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:23.978528 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:24.023075 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 20:56:24.026379 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:24.190721 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:24.230809 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:24.239348 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:24.307178 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:24.489614 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:24.608114 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:24.641444 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:24.652704 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:24.694000 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:24.739895 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:24.937685 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:24.941399 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:24.969839 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:24.975991 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 20:56:25.024976 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 20:56:25.090221 13317 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:56:25.090435 13317 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:56:25.091387 13317 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
W20240502 20:56:25.091919 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240502 20:56:25.092458 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:56:25.093442 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:56:25.094189 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:56:25.094429 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:25.094522 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:56:25.094676 13317 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:56:25.225600 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:25.264724 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:25.422153 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:25.423278 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:25.607538 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:25.735373 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:25.910280 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:26.035827 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:26.177824 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:26.217248 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:26.242767 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:26.315264 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:26.436764 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240502 20:56:26.454595 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:26.458973 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:26.511864 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:26.567070 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 20:56:26.582221 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:26.648433 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:26.883949 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:26.958545 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:27.034300 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:27.116232 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:27.168471 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:27.197775 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:27.213011 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:27.224162 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:27.286826 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:27.318480 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:27.354722 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:27.473991 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:27.630299 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:27.664268 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:27.731350 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 20:56:27.932884 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:28.124694 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:28.253460 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:28.414276 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:28.495882 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 20:56:28.496196 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
I20240502 20:56:28.513547 13847 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:28.513728 13847 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:56:28.514483 13847 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:56:28.514802 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:56:28.514976 27359 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:56:28.515964 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:56:28.516062 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:28.516152 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:56:28.516347 13847 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:28.610467 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:28.686329 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 20:56:28.701372 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:28.702858 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:28.765372 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:28.855346 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:28.896668 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:28.913465 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:28.960031 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:28.971022 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 20:56:28.999133 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:29.114604 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 20:56:29.179806 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:29.305330 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 20:56:29.513127 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:29.535910 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:29.590687 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:29.633431 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:29.682611 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:29.863543 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:29.866915 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:29.960772 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 20:56:30.025215 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240502 20:56:30.026278 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:30.048064 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 20:56:30.068784 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:30.072654 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 20:56:30.126739 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:30.233366 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:30.397708 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:30.856837 27158 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:56:30.910568 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:30.914919 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 20:56:30.927670 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:30.940346 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 20:56:31.123646 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:31.239733 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:31.265822 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:31.309768 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 20:56:31.371595 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20240502 20:56:31.447346 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 20:56:31.504772 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:31.504892 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:31.532692 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:31.573319 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 20:56:31.718802 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20240502 20:56:31.812645 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:31.855733 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:31.877951 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240502 20:56:32.068848 14292 consensus_queue.cc:579] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.186s)
I20240502 20:56:32.076476 27358 consensus_queue.cc:237] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:32.078706 27227 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 20:56:32.078933 27298 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): Couldn't send request to peer d1f3f8e444364d6faa42d651b5dbb997. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f00d12e425284c36ae8baed56c7cfaf6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:32.079190 14338 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 20:56:32.079381 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:56:32.079394 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:32.081941 14342 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:32.082510 27227 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:32.084594 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f00d12e425284c36ae8baed56c7cfaf6 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:32.085302 26802 catalog_manager.cc:5484] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 9 to 10, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:56:32.133546 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:32.169943 27297 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:56:32.333854 27302 tablet_copy_service.cc:454] P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-40df15a82a624913b3b712687d999134 on tablet 40df15a82a624913b3b712687d999134 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 has expired after 600.024s of idle time
I20240502 20:56:32.334000 27302 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-40df15a82a624913b3b712687d999134 on tablet 40df15a82a624913b3b712687d999134 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
I20240502 20:56:32.334038 27302 tablet_copy_service.cc:454] P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy session 1f40a11333e44ea0a34b6bc7618af522-6a8ce1df86914b2caf2d083d97eda4e6 on tablet 6a8ce1df86914b2caf2d083d97eda4e6 with peer 1f40a11333e44ea0a34b6bc7618af522 has expired after 600.095s of idle time
I20240502 20:56:32.334061 27302 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-6a8ce1df86914b2caf2d083d97eda4e6 on tablet 6a8ce1df86914b2caf2d083d97eda4e6 with peer 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:56:32.334082 27302 tablet_copy_service.cc:454] P ebd8300729ec4226acc38b9cff2d8d3b: tablet copy session 1f40a11333e44ea0a34b6bc7618af522-40df15a82a624913b3b712687d999134 on tablet 40df15a82a624913b3b712687d999134 with peer 1f40a11333e44ea0a34b6bc7618af522 has expired after 600.113s of idle time
I20240502 20:56:32.334116 27302 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session 1f40a11333e44ea0a34b6bc7618af522-40df15a82a624913b3b712687d999134 on tablet 40df15a82a624913b3b712687d999134 with peer 1f40a11333e44ea0a34b6bc7618af522
W20240502 20:56:32.379726 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:32.396418 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20240502 20:56:32.411944 14337 consensus_queue.cc:579] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 for more than 300 seconds (300.360s)
I20240502 20:56:32.419932 27227 consensus_queue.cc:237] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } }
I20240502 20:56:32.422178 27358 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:56:32.422317 26960 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 20:56:32.423091 13846 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 20:56:32.423363 14337 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 20:56:32.423836 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:56:32.426218 14337 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } }
I20240502 20:56:32.426975 27358 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } }
I20240502 20:56:32.428369 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd1b1316da2d47938164ac5b75f11bae with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:32.433905 26806 catalog_manager.cc:5484] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: UNKNOWN } } }
I20240502 20:56:32.436385 26959 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } }
W20240502 20:56:32.437631 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 20:56:32.438843 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cd1b1316da2d47938164ac5b75f11bae: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:32.439116 27227 consensus_queue.cc:237] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
W20240502 20:56:32.442606 27161 consensus_peers.cc:485] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 -> Peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): Couldn't send request to peer d1f3f8e444364d6faa42d651b5dbb997. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd1b1316da2d47938164ac5b75f11bae. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:32.442617 26959 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:56:32.443401 13846 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:56:32.443621 27358 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 20:56:32.444779 13846 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 20:56:32.446388 14337 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:32.448849 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cd1b1316da2d47938164ac5b75f11bae with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:32.449427 26959 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:32.450292 27358 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:32.452200 26802 catalog_manager.cc:5484] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 7 to 8, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:56:32.576167 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20240502 20:56:32.601864 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 20:56:32.617790 27295 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:56:32.621297 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:56:32.674137 14419 ts_tablet_manager.cc:902] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:56:32.674808 14419 tablet_copy_client.cc:308] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:56:32.675524 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet f00d12e425284c36ae8baed56c7cfaf6 from peer d1f3f8e444364d6faa42d651b5dbb997 ({username='slave'} at 127.19.66.130:43999)
I20240502 20:56:32.675655 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet f00d12e425284c36ae8baed56c7cfaf6 from peer d1f3f8e444364d6faa42d651b5dbb997 at {username='slave'} at 127.19.66.130:43999: session id = d1f3f8e444364d6faa42d651b5dbb997-f00d12e425284c36ae8baed56c7cfaf6
I20240502 20:56:32.676743 27382 tablet_copy_source_session.cc:205] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:32.678849 14419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00d12e425284c36ae8baed56c7cfaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:32.682706 14419 tablet_copy_client.cc:783] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:32.690562 14419 tablet_copy_client.cc:647] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 1 WAL segments...
I20240502 20:56:32.692296 14419 tablet_copy_client.cc:523] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:56:32.694099 14419 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:56:32.703374 14419 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:32.703794 14419 tablet_bootstrap.cc:492] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap complete.
I20240502 20:56:32.704028 14419 ts_tablet_manager.cc:1367] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.003s
I20240502 20:56:32.704579 14419 raft_consensus.cc:348] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:32.704713 14419 raft_consensus.cc:729] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: LEARNER
I20240502 20:56:32.704805 14419 consensus_queue.cc:260] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:32.705178 14419 ts_tablet_manager.cc:1398] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240502 20:56:32.706543 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session d1f3f8e444364d6faa42d651b5dbb997-f00d12e425284c36ae8baed56c7cfaf6 received from {username='slave'} at 127.19.66.130:43999
I20240502 20:56:32.706676 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session d1f3f8e444364d6faa42d651b5dbb997-f00d12e425284c36ae8baed56c7cfaf6 on tablet f00d12e425284c36ae8baed56c7cfaf6 with peer d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:56:32.718943 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:56:32.735999 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240502 20:56:32.834718 14419 ts_tablet_manager.cc:902] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:56:32.835778 14419 tablet_copy_client.cc:308] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:56:32.836290 27248 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet cd1b1316da2d47938164ac5b75f11bae from peer d1f3f8e444364d6faa42d651b5dbb997 ({username='slave'} at 127.19.66.130:60245)
I20240502 20:56:32.836405 27248 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet cd1b1316da2d47938164ac5b75f11bae from peer d1f3f8e444364d6faa42d651b5dbb997 at {username='slave'} at 127.19.66.130:60245: session id = d1f3f8e444364d6faa42d651b5dbb997-cd1b1316da2d47938164ac5b75f11bae
I20240502 20:56:32.838317 27248 tablet_copy_source_session.cc:205] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:32.839751 14419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1b1316da2d47938164ac5b75f11bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:32.842511 14419 tablet_copy_client.cc:783] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:32.851168 14419 tablet_copy_client.cc:647] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 1 WAL segments...
I20240502 20:56:32.853412 14419 tablet_copy_client.cc:523] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 20:56:32.854786 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 20:56:32.855484 14419 tablet_bootstrap.cc:492] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:56:32.867751 14419 tablet_bootstrap.cc:492] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:32.868273 14419 tablet_bootstrap.cc:492] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap complete.
I20240502 20:56:32.868559 14419 ts_tablet_manager.cc:1367] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.013s	user 0.012s	sys 0.000s
I20240502 20:56:32.869320 14419 raft_consensus.cc:348] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:32.869525 14419 raft_consensus.cc:729] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: LEARNER
I20240502 20:56:32.869689 14419 consensus_queue.cc:260] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:32.870095 14419 ts_tablet_manager.cc:1398] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.003s	sys 0.000s
I20240502 20:56:32.871088 27248 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session d1f3f8e444364d6faa42d651b5dbb997-cd1b1316da2d47938164ac5b75f11bae received from {username='slave'} at 127.19.66.130:60245
I20240502 20:56:32.871229 27248 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session d1f3f8e444364d6faa42d651b5dbb997-cd1b1316da2d47938164ac5b75f11bae on tablet cd1b1316da2d47938164ac5b75f11bae with peer d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:56:32.907047 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(f00d12e425284c36ae8baed56c7cfaf6): 4840 bytes on disk
I20240502 20:56:32.907603 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(f00d12e425284c36ae8baed56c7cfaf6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 20:56:32.908461 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(cd1b1316da2d47938164ac5b75f11bae): 4586 bytes on disk
I20240502 20:56:32.908882 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(cd1b1316da2d47938164ac5b75f11bae) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240502 20:56:32.977547 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:56:33.103765 14338 consensus_queue.cc:579] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.014s)
I20240502 20:56:33.108417 27358 consensus_queue.cc:237] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:33.110743 26959 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 20:56:33.111213 27298 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): Couldn't send request to peer c107c43e538d48f58625c4320f0aa4b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a8ce1df86914b2caf2d083d97eda4e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:33.111590 14473 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:56:33.111775 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:56:33.113294 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:33.114028 14475 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:33.114609 26959 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:33.117655 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6a8ce1df86914b2caf2d083d97eda4e6 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:33.118415 26807 catalog_manager.cc:5484] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 10 to 11, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:56:33.162083 26893 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:56:33.243748 27090 raft_consensus.cc:1203] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
W20240502 20:56:33.296444 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20240502 20:56:33.326823 27090 raft_consensus.cc:1203] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20240502 20:56:33.510028 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 20:56:33.550487 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20240502 20:56:33.664032 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cd1b1316da2d47938164ac5b75f11bae: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:33.676947 14468 raft_consensus.cc:1050] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 to VOTER
I20240502 20:56:33.677552 14468 consensus_queue.cc:237] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:56:33.679420 27090 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:56:33.679611 27227 raft_consensus.cc:1261] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:56:33.679883 14529 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:56:33.680151 14528 consensus_queue.cc:1035] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 20:56:33.680305 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:56:33.680311 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:33.682907 14529 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.683491 27227 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.685361 27090 raft_consensus.cc:2932] T f00d12e425284c36ae8baed56c7cfaf6 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.686543 26807 catalog_manager.cc:5484] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 10 to 11, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:33.703581 14537 ts_tablet_manager.cc:902] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:56:33.704219 14537 tablet_copy_client.cc:308] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:56:33.704954 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 6a8ce1df86914b2caf2d083d97eda4e6 from peer c107c43e538d48f58625c4320f0aa4b8 ({username='slave'} at 127.19.66.131:37981)
I20240502 20:56:33.705308 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 6a8ce1df86914b2caf2d083d97eda4e6 from peer c107c43e538d48f58625c4320f0aa4b8 at {username='slave'} at 127.19.66.131:37981: session id = c107c43e538d48f58625c4320f0aa4b8-6a8ce1df86914b2caf2d083d97eda4e6
W20240502 20:56:33.706760 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20240502 20:56:33.706818 27382 tablet_copy_source_session.cc:205] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:33.708478 14537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a8ce1df86914b2caf2d083d97eda4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:33.710850 14537 tablet_copy_client.cc:783] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:33.717051 14537 tablet_copy_client.cc:647] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 1 WAL segments...
W20240502 20:56:33.718591 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:56:33.718953 14537 tablet_copy_client.cc:523] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:56:33.720718 14537 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:56:33.730535 14537 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:33.730888 14537 tablet_bootstrap.cc:492] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap complete.
I20240502 20:56:33.731107 14537 ts_tablet_manager.cc:1367] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.001s
I20240502 20:56:33.731628 14537 raft_consensus.cc:348] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:33.731781 14537 raft_consensus.cc:729] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: LEARNER
I20240502 20:56:33.731911 14537 consensus_queue.cc:260] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:33.732208 14537 ts_tablet_manager.cc:1398] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:56:33.733234 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session c107c43e538d48f58625c4320f0aa4b8-6a8ce1df86914b2caf2d083d97eda4e6 received from {username='slave'} at 127.19.66.131:37981
I20240502 20:56:33.733387 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session c107c43e538d48f58625c4320f0aa4b8-6a8ce1df86914b2caf2d083d97eda4e6 on tablet 6a8ce1df86914b2caf2d083d97eda4e6 with peer c107c43e538d48f58625c4320f0aa4b8
I20240502 20:56:33.747943 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(6a8ce1df86914b2caf2d083d97eda4e6): 4840 bytes on disk
I20240502 20:56:33.748618 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(6a8ce1df86914b2caf2d083d97eda4e6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240502 20:56:33.841240 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 20:56:33.877604 26892 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:56:33.923797 14397 raft_consensus.cc:1050] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 to VOTER
I20240502 20:56:33.924289 14397 consensus_queue.cc:237] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:56:33.926231 26959 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:56:33.926275 27090 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:56:33.926673 27358 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 20:56:33.926766 14563 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:56:33.927031 14397 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:56:33.927206 14567 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 20:56:33.930946 14563 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.931437 26959 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.931883 27090 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.935111 26802 catalog_manager.cc:5484] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 8 to 9, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.936460 27358 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.942248 27227 consensus_queue.cc:237] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:56:33.943653 27090 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:56:33.943941 26959 raft_consensus.cc:1261] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:56:33.944419 14563 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:56:33.944690 14566 consensus_queue.cc:1035] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:56:33.946621 14397 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.947113 26959 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.947085 27090 raft_consensus.cc:2932] T cd1b1316da2d47938164ac5b75f11bae P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.948736 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd1b1316da2d47938164ac5b75f11bae with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:33.950073 26805 catalog_manager.cc:5484] T cd1b1316da2d47938164ac5b75f11bae P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 9 to 10, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:33.954046 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet cd1b1316da2d47938164ac5b75f11bae with delete_type TABLET_DATA_TOMBSTONED (TS ebd8300729ec4226acc38b9cff2d8d3b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34124
I20240502 20:56:33.954720 14571 tablet_replica.cc:330] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b: stopping tablet replica
I20240502 20:56:33.955230 14571 raft_consensus.cc:2229] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:56:33.955677 14571 raft_consensus.cc:2258] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:56:33.956957 14571 ts_tablet_manager.cc:1875] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 20:56:33.959546 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240502 20:56:33.961980 14571 ts_tablet_manager.cc:1888] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 20:56:33.962097 14571 log.cc:1196] T cd1b1316da2d47938164ac5b75f11bae P ebd8300729ec4226acc38b9cff2d8d3b: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/wals/cd1b1316da2d47938164ac5b75f11bae
I20240502 20:56:33.962891 26796 catalog_manager.cc:4830] TS ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): tablet cd1b1316da2d47938164ac5b75f11bae (table rebalance_test_table_7 [id=f67895aa93e54fa6a76a86a676d1a1ff]) successfully deleted
W20240502 20:56:34.094339 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
I20240502 20:56:34.100735 14529 consensus_queue.cc:579] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.316s)
I20240502 20:56:34.104221 27358 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:56:34.106422 27227 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:34.106422 26959 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:34.107043 14529 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:34.107297 14479 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:34.110747 14528 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:34.111359 26959 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:34.111498 27227 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:34.112704 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5cf737e20ca74db1b94ccff5d60133ee with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:34.114259 26806 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 12 to 13, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:34.118693 27358 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:34.121019 26959 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:34.121374 27227 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240502 20:56:34.121564 27298 consensus_peers.cc:485] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b -> Peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): Couldn't send request to peer d1f3f8e444364d6faa42d651b5dbb997. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:34.122092 14528 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:34.122435 14468 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:34.124405 14479 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:34.124996 26959 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:34.126430 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5cf737e20ca74db1b94ccff5d60133ee with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:34.126578 27227 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:34.128264 26806 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 13 to 14, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 20:56:34.135699 27159 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 20:56:34.181147 27296 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:34.196238 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240502 20:56:34.207031 14503 consensus_queue.cc:579] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.116s)
I20240502 20:56:34.215553 26959 consensus_queue.cc:237] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:34.217968 27227 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 20:56:34.218482 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:34.218515 27358 raft_consensus.cc:1261] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:56:34.218597 14601 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:56:34.219063 14503 consensus_queue.cc:1035] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:56:34.222222 14601 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.222808 27227 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.225608 26794 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2b61909194d488682afd91c0f796405 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:34.226781 26805 catalog_manager.cc:5484] T a2b61909194d488682afd91c0f796405 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.227632 27358 raft_consensus.cc:2932] T a2b61909194d488682afd91c0f796405 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.249816 27227 raft_consensus.cc:1203] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20240502 20:56:34.276118 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20240502 20:56:34.575651 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:34.646886 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20240502 20:56:34.654299 14648 ts_tablet_manager.cc:902] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:56:34.655385 14648 tablet_copy_client.cc:308] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:56:34.656360 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 5cf737e20ca74db1b94ccff5d60133ee from peer d1f3f8e444364d6faa42d651b5dbb997 ({username='slave'} at 127.19.66.130:43999)
I20240502 20:56:34.656504 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 5cf737e20ca74db1b94ccff5d60133ee from peer d1f3f8e444364d6faa42d651b5dbb997 at {username='slave'} at 127.19.66.130:43999: session id = d1f3f8e444364d6faa42d651b5dbb997-5cf737e20ca74db1b94ccff5d60133ee
I20240502 20:56:34.657738 27382 tablet_copy_source_session.cc:205] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:34.659073 14648 ts_tablet_manager.cc:1875] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:56:34.660315 14479 raft_consensus.cc:1050] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER c107c43e538d48f58625c4320f0aa4b8 to VOTER
I20240502 20:56:34.661000 14479 consensus_queue.cc:237] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:34.662304 14648 ts_tablet_manager.cc:1888] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 20:56:34.662533 14648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf737e20ca74db1b94ccff5d60133ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:34.663447 27227 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240502 20:56:34.663803 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:34.664027 14648 tablet_copy_client.cc:783] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:34.664376 14655 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 20:56:34.665215 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:34.666429 26959 raft_consensus.cc:1261] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:56:34.667106 14655 consensus_queue.cc:1035] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:34.671937 14654 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.673035 26959 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.673590 14648 tablet_copy_client.cc:647] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 1 WAL segments...
I20240502 20:56:34.674805 27227 raft_consensus.cc:2932] T 6a8ce1df86914b2caf2d083d97eda4e6 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.677563 14648 tablet_copy_client.cc:523] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:56:34.677405 26806 catalog_manager.cc:5484] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 11 to 12, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:34.679603 14648 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:56:34.691656 14648 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:34.692157 14648 tablet_bootstrap.cc:492] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap complete.
I20240502 20:56:34.692409 14648 ts_tablet_manager.cc:1367] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.013s	user 0.006s	sys 0.004s
I20240502 20:56:34.693042 14648 raft_consensus.cc:348] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:34.693259 14648 raft_consensus.cc:729] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: LEARNER
I20240502 20:56:34.693408 14648 consensus_queue.cc:260] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:34.693787 14648 ts_tablet_manager.cc:1398] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.004s
I20240502 20:56:34.694939 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session d1f3f8e444364d6faa42d651b5dbb997-5cf737e20ca74db1b94ccff5d60133ee received from {username='slave'} at 127.19.66.130:43999
I20240502 20:56:34.695101 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session d1f3f8e444364d6faa42d651b5dbb997-5cf737e20ca74db1b94ccff5d60133ee on tablet 5cf737e20ca74db1b94ccff5d60133ee with peer d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:56:34.708537 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20240502 20:56:34.712970 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cd1b1316da2d47938164ac5b75f11bae: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:56:34.714018 27160 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 20:56:34.780130 27024 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:56:34.796725 14655 consensus_queue.cc:579] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 for more than 300 seconds (300.377s)
I20240502 20:56:34.800263 27358 consensus_queue.cc:237] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:34.801859 26959 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:56:34.801859 27227 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 20:56:34.802451 14655 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:56:34.802630 14528 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 20:56:34.804759 14528 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.805423 27227 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.806136 26959 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.806718 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40df15a82a624913b3b712687d999134 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 20:56:34.808346 27024 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 20:56:34.809228 26807 catalog_manager.cc:5484] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 9 to 10, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:34.815101 27358 consensus_queue.cc:237] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:34.817363 27227 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 20:56:34.817672 27298 consensus_peers.cc:485] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): Couldn't send request to peer d1f3f8e444364d6faa42d651b5dbb997. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:34.817696 26959 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:56:34.818009 14528 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:56:34.818534 14654 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:56:34.820742 14655 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:34.821413 27227 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:34.821413 26959 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } } }
I20240502 20:56:34.823549 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 40df15a82a624913b3b712687d999134 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:34.824642 26806 catalog_manager.cc:5484] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 10 to 11, NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:56:34.921353 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(5cf737e20ca74db1b94ccff5d60133ee): 4586 bytes on disk
I20240502 20:56:34.921967 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(5cf737e20ca74db1b94ccff5d60133ee) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240502 20:56:34.922932 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 20:56:35.036510 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20240502 20:56:35.065246 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 20:56:35.110134 26892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 20:56:35.191788 14682 consensus_queue.cc:579] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 for more than 300 seconds (300.188s)
I20240502 20:56:35.195137 27090 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:56:35.197268 26960 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240502 20:56:35.197650 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:35.197798 14682 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:35.197996 27358 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:56:35.199024 14101 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:35.200037 14720 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:35.200562 26960 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:35.201864 27358 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:35.204293 26807 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 11 to 12, NON_VOTER 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:35.204962 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1b9ff3c921d474fa5ad106d1c17ab00 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:35.208662 27090 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:35.211176 26960 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:35.211619 14101 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 20:56:35.211802 27027 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): Couldn't send request to peer c107c43e538d48f58625c4320f0aa4b8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:35.212425 27358 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240502 20:56:35.212484 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:35.213260 14101 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:35.213732 14682 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.215567 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d1b9ff3c921d474fa5ad106d1c17ab00 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:35.215934 26960 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.217411 27090 raft_consensus.cc:1203] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14]   Dedup: 1.14->[]
I20240502 20:56:35.219949 26806 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 12 to 13, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.226104 14720 consensus_queue.cc:579] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.325s)
I20240502 20:56:35.226248 27358 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.230610 27090 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } }
I20240502 20:56:35.232513 26960 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:35.233000 14723 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:35.233261 27358 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:35.233836 14723 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:35.236218 14720 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:56:35.237386 27358 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:56:35.238258 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f00c4b7954564820875a25da5834416a with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:35.240265 26805 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 12 to 13, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:35.241376 26960 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:56:35.249671 27090 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:35.251832 27361 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:35.252310 14101 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:35.252461 26960 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240502 20:56:35.252708 27027 consensus_peers.cc:485] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 -> Peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): Couldn't send request to peer c107c43e538d48f58625c4320f0aa4b8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f00c4b7954564820875a25da5834416a. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:35.253415 14101 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:35.254967 14720 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.257500 27361 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.257742 26960 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.258131 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f00c4b7954564820875a25da5834416a with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:35.262076 26806 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 13 to 14, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.284889 14654 consensus_queue.cc:579] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.172s)
I20240502 20:56:35.287935 27361 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:56:35.290179 26960 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:56:35.290724 14479 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:56:35.290864 27090 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:56:35.291363 14479 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:56:35.292956 14654 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:35.293550 26960 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:35.294234 27090 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:35.295255 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5586b9e60b544aed87c24f7b3ce0198c with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:35.296826 26805 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:35.300861 27361 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:35.302623 26960 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240502 20:56:35.302893 27298 consensus_peers.cc:485] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b -> Peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): Couldn't send request to peer c107c43e538d48f58625c4320f0aa4b8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:35.303081 14528 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:35.303611 27090 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:56:35.304060 14528 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:35.305473 14479 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.305917 26960 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.306648 27090 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.307910 26796 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5586b9e60b544aed87c24f7b3ce0198c with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:35.309862 26807 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 11 to 12, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:56:35.339490 14756 ts_tablet_manager.cc:902] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:56:35.340168 14756 tablet_copy_client.cc:308] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:56:35.340739 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 40df15a82a624913b3b712687d999134 from peer d1f3f8e444364d6faa42d651b5dbb997 ({username='slave'} at 127.19.66.130:43999)
I20240502 20:56:35.340888 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 40df15a82a624913b3b712687d999134 from peer d1f3f8e444364d6faa42d651b5dbb997 at {username='slave'} at 127.19.66.130:43999: session id = d1f3f8e444364d6faa42d651b5dbb997-40df15a82a624913b3b712687d999134
I20240502 20:56:35.342183 27382 tablet_copy_source_session.cc:205] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:35.343303 14756 ts_tablet_manager.cc:1875] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:56:35.346197 14756 ts_tablet_manager.cc:1888] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 20:56:35.346431 14756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40df15a82a624913b3b712687d999134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:35.347759 14756 tablet_copy_client.cc:783] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:35.354300 14756 tablet_copy_client.cc:647] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Starting download of 1 WAL segments...
I20240502 20:56:35.356165 14756 tablet_copy_client.cc:523] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:56:35.357829 14756 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap starting.
I20240502 20:56:35.366789 14756 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:35.367174 14756 tablet_bootstrap.cc:492] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Bootstrap complete.
I20240502 20:56:35.367400 14756 ts_tablet_manager.cc:1367] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.002s
I20240502 20:56:35.367902 14756 raft_consensus.cc:348] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:35.368064 14756 raft_consensus.cc:729] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f3f8e444364d6faa42d651b5dbb997, State: Initialized, Role: LEARNER
I20240502 20:56:35.368180 14756 consensus_queue.cc:260] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: NON_VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: true } }
I20240502 20:56:35.368619 14756 ts_tablet_manager.cc:1398] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.001s
I20240502 20:56:35.369453 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session d1f3f8e444364d6faa42d651b5dbb997-40df15a82a624913b3b712687d999134 received from {username='slave'} at 127.19.66.130:43999
I20240502 20:56:35.369553 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session d1f3f8e444364d6faa42d651b5dbb997-40df15a82a624913b3b712687d999134 on tablet 40df15a82a624913b3b712687d999134 with peer d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:56:35.378929 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
I20240502 20:56:35.426446 27141 maintenance_manager.cc:419] P d1f3f8e444364d6faa42d651b5dbb997: Scheduling UndoDeltaBlockGCOp(40df15a82a624913b3b712687d999134): 4586 bytes on disk
I20240502 20:56:35.427099 27028 maintenance_manager.cc:643] P d1f3f8e444364d6faa42d651b5dbb997: UndoDeltaBlockGCOp(40df15a82a624913b3b712687d999134) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 20:56:35.588472 14607 consensus_queue.cc:579] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.341s)
I20240502 20:56:35.593134 26960 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } }
I20240502 20:56:35.594933 27361 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:56:35.594938 27090 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 20:56:35.595432 14773 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:56:35.595633 14772 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 20:56:35.598634 14607 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:56:35.599479 27090 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:56:35.600744 26794 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af693f5816e046f592fbd9b2e7ef0ad9 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:35.601569 27361 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:56:35.601982 26805 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 10 to 11, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 20:56:35.604837 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 20:56:35.607113 26960 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:35.609804 27361 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:56:35.609990 27090 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:56:35.611152 14772 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 20:56:35.611830 26894 consensus_peers.cc:485] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 -> Peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): Couldn't send request to peer c107c43e538d48f58625c4320f0aa4b8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:35.611948 14772 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:35.613662 14607 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.614643 27361 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.614591 27090 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.615837 26794 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet af693f5816e046f592fbd9b2e7ef0ad9 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:35.619047 26807 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 11 to 12, NON_VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) added. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } } }
I20240502 20:56:35.633167 14777 ts_tablet_manager.cc:902] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:56:35.633797 14777 tablet_copy_client.cc:308] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:56:35.634668 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet d1b9ff3c921d474fa5ad106d1c17ab00 from peer c107c43e538d48f58625c4320f0aa4b8 ({username='slave'} at 127.19.66.131:40809)
I20240502 20:56:35.634807 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet d1b9ff3c921d474fa5ad106d1c17ab00 from peer c107c43e538d48f58625c4320f0aa4b8 at {username='slave'} at 127.19.66.131:40809: session id = c107c43e538d48f58625c4320f0aa4b8-d1b9ff3c921d474fa5ad106d1c17ab00
I20240502 20:56:35.636827 27112 tablet_copy_source_session.cc:205] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:35.637938 14777 ts_tablet_manager.cc:1875] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:56:35.640333 14782 ts_tablet_manager.cc:902] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8: Initiating tablet copy from peer d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307)
I20240502 20:56:35.641324 14782 tablet_copy_client.cc:308] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.130:43307
I20240502 20:56:35.641618 14777 ts_tablet_manager.cc:1888] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:56:35.641847 14777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b9ff3c921d474fa5ad106d1c17ab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:35.642388 27112 tablet_copy_service.cc:140] P d1f3f8e444364d6faa42d651b5dbb997: Received BeginTabletCopySession request for tablet f00c4b7954564820875a25da5834416a from peer c107c43e538d48f58625c4320f0aa4b8 ({username='slave'} at 127.19.66.131:40809)
I20240502 20:56:35.642572 27112 tablet_copy_service.cc:161] P d1f3f8e444364d6faa42d651b5dbb997: Beginning new tablet copy session on tablet f00c4b7954564820875a25da5834416a from peer c107c43e538d48f58625c4320f0aa4b8 at {username='slave'} at 127.19.66.131:40809: session id = c107c43e538d48f58625c4320f0aa4b8-f00c4b7954564820875a25da5834416a
I20240502 20:56:35.643796 14777 tablet_copy_client.cc:783] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:35.644357 27112 tablet_copy_source_session.cc:205] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:35.645951 14782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00c4b7954564820875a25da5834416a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:35.649026 14782 tablet_copy_client.cc:783] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:35.651268 14777 tablet_copy_client.cc:647] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 1 WAL segments...
I20240502 20:56:35.653098 14777 tablet_copy_client.cc:523] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:56:35.654953 14777 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:56:35.657455 14782 tablet_copy_client.cc:647] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 1 WAL segments...
I20240502 20:56:35.659204 14782 tablet_copy_client.cc:523] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:56:35.661196 14782 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:56:35.668740 14777 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:35.669253 14777 tablet_bootstrap.cc:492] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap complete.
I20240502 20:56:35.669498 14777 ts_tablet_manager.cc:1367] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.015s	user 0.014s	sys 0.000s
I20240502 20:56:35.670239 14777 raft_consensus.cc:348] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:35.670636 14777 raft_consensus.cc:729] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: LEARNER
I20240502 20:56:35.670868 14777 consensus_queue.cc:260] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:35.671620 14777 ts_tablet_manager.cc:1398] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 20:56:35.672518 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session c107c43e538d48f58625c4320f0aa4b8-d1b9ff3c921d474fa5ad106d1c17ab00 received from {username='slave'} at 127.19.66.131:40809
I20240502 20:56:35.672649 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session c107c43e538d48f58625c4320f0aa4b8-d1b9ff3c921d474fa5ad106d1c17ab00 on tablet d1b9ff3c921d474fa5ad106d1c17ab00 with peer c107c43e538d48f58625c4320f0aa4b8
I20240502 20:56:35.676959 14529 raft_consensus.cc:1050] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 to VOTER
I20240502 20:56:35.677817 14529 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:56:35.677918 14782 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:35.678328 14782 tablet_bootstrap.cc:492] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8: Bootstrap complete.
I20240502 20:56:35.678557 14782 ts_tablet_manager.cc:1367] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.018s	user 0.016s	sys 0.001s
I20240502 20:56:35.679106 14782 raft_consensus.cc:348] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:35.679270 14782 raft_consensus.cc:729] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: LEARNER
I20240502 20:56:35.679379 14782 consensus_queue.cc:260] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:35.679778 14782 ts_tablet_manager.cc:1398] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:56:35.680073 26960 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 20:56:35.680125 27090 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 20:56:35.680595 14479 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 20:56:35.680687 27227 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 20:56:35.680935 14654 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 20:56:35.682483 27112 tablet_copy_service.cc:342] P d1f3f8e444364d6faa42d651b5dbb997: Request end of tablet copy session c107c43e538d48f58625c4320f0aa4b8-f00c4b7954564820875a25da5834416a received from {username='slave'} at 127.19.66.131:40809
I20240502 20:56:35.682587 27112 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session c107c43e538d48f58625c4320f0aa4b8-f00c4b7954564820875a25da5834416a on tablet f00c4b7954564820875a25da5834416a with peer c107c43e538d48f58625c4320f0aa4b8
I20240502 20:56:35.683497 14654 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.685050 14654 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 20:56:35.686259 26960 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.687099 27227 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.688953 26805 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 14 to 15, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:35.691040 27090 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.695703 27361 consensus_queue.cc:237] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:56:35.697664 26960 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240502 20:56:35.697748 27090 raft_consensus.cc:1261] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240502 20:56:35.698135 14479 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 20:56:35.698454 14479 consensus_queue.cc:1035] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 20:56:35.699949 14479 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.700613 26960 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.700851 27090 raft_consensus.cc:2932] T 5cf737e20ca74db1b94ccff5d60133ee P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.701951 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5cf737e20ca74db1b94ccff5d60133ee with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:35.703647 26805 catalog_manager.cc:5484] T 5cf737e20ca74db1b94ccff5d60133ee P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 15 to 16, VOTER c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:35.707584 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf737e20ca74db1b94ccff5d60133ee with delete_type TABLET_DATA_TOMBSTONED (TS c107c43e538d48f58625c4320f0aa4b8 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:40862
I20240502 20:56:35.708097 14794 tablet_replica.cc:330] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8: stopping tablet replica
I20240502 20:56:35.708288 14794 raft_consensus.cc:2229] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:56:35.708478 14794 raft_consensus.cc:2258] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:56:35.709347 14794 ts_tablet_manager.cc:1875] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:56:35.714594 14794 ts_tablet_manager.cc:1888] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.15
I20240502 20:56:35.714748 14794 log.cc:1196] T 5cf737e20ca74db1b94ccff5d60133ee P c107c43e538d48f58625c4320f0aa4b8: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-2/wal/wals/5cf737e20ca74db1b94ccff5d60133ee
I20240502 20:56:35.715584 26797 catalog_manager.cc:4830] TS c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123): tablet 5cf737e20ca74db1b94ccff5d60133ee (table rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]) successfully deleted
I20240502 20:56:35.759570 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(d1b9ff3c921d474fa5ad106d1c17ab00): 4925 bytes on disk
I20240502 20:56:35.760322 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(d1b9ff3c921d474fa5ad106d1c17ab00) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 20:56:35.761593 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(f00c4b7954564820875a25da5834416a): 4925 bytes on disk
I20240502 20:56:35.762147 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(f00c4b7954564820875a25da5834416a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240502 20:56:35.769779 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet af693f5816e046f592fbd9b2e7ef0ad9: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:56:35.775930 14782 ts_tablet_manager.cc:902] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Initiating tablet copy from peer ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007)
I20240502 20:56:35.777145 14782 tablet_copy_client.cc:308] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.132:41007
I20240502 20:56:35.777751 27382 tablet_copy_service.cc:140] P ebd8300729ec4226acc38b9cff2d8d3b: Received BeginTabletCopySession request for tablet 5586b9e60b544aed87c24f7b3ce0198c from peer c107c43e538d48f58625c4320f0aa4b8 ({username='slave'} at 127.19.66.131:37981)
I20240502 20:56:35.777896 27382 tablet_copy_service.cc:161] P ebd8300729ec4226acc38b9cff2d8d3b: Beginning new tablet copy session on tablet 5586b9e60b544aed87c24f7b3ce0198c from peer c107c43e538d48f58625c4320f0aa4b8 at {username='slave'} at 127.19.66.131:37981: session id = c107c43e538d48f58625c4320f0aa4b8-5586b9e60b544aed87c24f7b3ce0198c
I20240502 20:56:35.778865 27382 tablet_copy_source_session.cc:205] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:35.779670 14782 ts_tablet_manager.cc:1875] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:56:35.781935 14790 consensus_queue.cc:579] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 6706d1c0946d42a9bae62d719e1e5c31 for more than 300 seconds (300.424s)
I20240502 20:56:35.782011 14782 ts_tablet_manager.cc:1888] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:56:35.782230 14782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5586b9e60b544aed87c24f7b3ce0198c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:35.783598 14782 tablet_copy_client.cc:783] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:35.785144 27227 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:56:35.786768 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 20:56:35.787009 27091 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 20:56:35.787609 14813 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 20:56:35.787935 14790 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 20:56:35.790310 14790 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.790845 27361 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
W20240502 20:56:35.791723 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
I20240502 20:56:35.792461 27091 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.792806 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bf189c19164442e5af0827f1fb7dc085 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:35.793249 14529 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:56:35.793355 14529 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:56:35.793844 14529 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
I20240502 20:56:35.794025 26805 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 14 to 15, VOTER 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.795197 14782 tablet_copy_client.cc:647] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 1 WAL segments...
I20240502 20:56:35.796883 14782 tablet_copy_client.cc:523] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 20:56:35.798159 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:56:35.798272 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:35.798355 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:56:35.798820 14782 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:56:35.799026 14529 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:56:35.803395 27227 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:56:35.805122 27091 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
W20240502 20:56:35.805286 27158 consensus_peers.cc:485] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf189c19164442e5af0827f1fb7dc085. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:35.805569 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240502 20:56:35.805560 14813 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 20:56:35.806111 14813 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 20:56:35.807581 14563 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:56:35.808135 27091 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:56:35.808485 27361 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:56:35.809791 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bf189c19164442e5af0827f1fb7dc085 with cas_config_opid_index 15: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:35.811506 26807 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 15 to 16, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:56:35.813026 14790 consensus_queue.cc:579] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Leader has been unable to successfully communicate with peer 1bc8a142cfc7487bacc573afb0a92b74 for more than 300 seconds (300.327s)
I20240502 20:56:35.813568 14782 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:35.813877 14782 tablet_bootstrap.cc:492] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Bootstrap complete.
I20240502 20:56:35.814092 14782 ts_tablet_manager.cc:1367] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.015s	user 0.009s	sys 0.004s
I20240502 20:56:35.814647 14782 raft_consensus.cc:348] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:35.814770 14782 raft_consensus.cc:729] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: LEARNER
I20240502 20:56:35.814880 14782 consensus_queue.cc:260] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:35.815254 14782 ts_tablet_manager.cc:1398] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:56:35.816938 27382 tablet_copy_service.cc:342] P ebd8300729ec4226acc38b9cff2d8d3b: Request end of tablet copy session c107c43e538d48f58625c4320f0aa4b8-5586b9e60b544aed87c24f7b3ce0198c received from {username='slave'} at 127.19.66.131:37981
I20240502 20:56:35.817036 27382 tablet_copy_service.cc:433] P ebd8300729ec4226acc38b9cff2d8d3b: ending tablet copy session c107c43e538d48f58625c4320f0aa4b8-5586b9e60b544aed87c24f7b3ce0198c on tablet 5586b9e60b544aed87c24f7b3ce0198c with peer c107c43e538d48f58625c4320f0aa4b8
I20240502 20:56:35.818888 27227 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:56:35.820323 27361 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:56:35.820775 27091 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:56:35.820780 14819 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:35.821295 14819 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:35.823508 14813 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.824121 27361 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.824282 27091 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.826725 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a906569ab3b14520b4876bc8d5450f7d with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:35.828147 26806 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 11 to 12, VOTER 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135) evicted. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:35.833462 27227 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:56:35.835270 27361 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240502 20:56:35.835899 27158 consensus_peers.cc:485] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 -> Peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): Couldn't send request to peer 297de823852d4a8da44791ab03131cd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a906569ab3b14520b4876bc8d5450f7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:35.835959 27091 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:35.836537 14790 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:35.837253 14790 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:35.838786 14563 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:56:35.839396 27361 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
I20240502 20:56:35.840400 26797 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a906569ab3b14520b4876bc8d5450f7d with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 20:56:35.843006 26807 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 12 to 13, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 20:56:35.843394 27091 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } } }
W20240502 20:56:35.876178 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
I20240502 20:56:35.920667 27091 raft_consensus.cc:1203] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
I20240502 20:56:36.014237 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(5586b9e60b544aed87c24f7b3ce0198c): 4840 bytes on disk
I20240502 20:56:36.014820 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(5586b9e60b544aed87c24f7b3ce0198c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 20:56:36.020992 14782 ts_tablet_manager.cc:902] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Initiating tablet copy from peer 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:56:36.021589 14782 tablet_copy_client.cc:308] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.129:38835
I20240502 20:56:36.022181 26980 tablet_copy_service.cc:140] P 297de823852d4a8da44791ab03131cd3: Received BeginTabletCopySession request for tablet af693f5816e046f592fbd9b2e7ef0ad9 from peer c107c43e538d48f58625c4320f0aa4b8 ({username='slave'} at 127.19.66.131:58105)
I20240502 20:56:36.022300 26980 tablet_copy_service.cc:161] P 297de823852d4a8da44791ab03131cd3: Beginning new tablet copy session on tablet af693f5816e046f592fbd9b2e7ef0ad9 from peer c107c43e538d48f58625c4320f0aa4b8 at {username='slave'} at 127.19.66.131:58105: session id = c107c43e538d48f58625c4320f0aa4b8-af693f5816e046f592fbd9b2e7ef0ad9
I20240502 20:56:36.023365 26980 tablet_copy_source_session.cc:205] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:36.025051 14782 ts_tablet_manager.cc:1875] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 20:56:36.027664 14782 ts_tablet_manager.cc:1888] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 20:56:36.028106 14782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af693f5816e046f592fbd9b2e7ef0ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:36.029940 14782 tablet_copy_client.cc:783] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:36.038298 14782 tablet_copy_client.cc:647] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Starting download of 1 WAL segments...
I20240502 20:56:36.040259 14782 tablet_copy_client.cc:523] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:56:36.041922 14782 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap starting.
I20240502 20:56:36.051268 14782 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:36.051610 14782 tablet_bootstrap.cc:492] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Bootstrap complete.
I20240502 20:56:36.051827 14782 ts_tablet_manager.cc:1367] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240502 20:56:36.052342 14782 raft_consensus.cc:348] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:36.052481 14782 raft_consensus.cc:729] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c107c43e538d48f58625c4320f0aa4b8, State: Initialized, Role: LEARNER
I20240502 20:56:36.052628 14782 consensus_queue.cc:260] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: NON_VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: true } }
I20240502 20:56:36.052909 14782 ts_tablet_manager.cc:1398] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 20:56:36.053658 26980 tablet_copy_service.cc:342] P 297de823852d4a8da44791ab03131cd3: Request end of tablet copy session c107c43e538d48f58625c4320f0aa4b8-af693f5816e046f592fbd9b2e7ef0ad9 received from {username='slave'} at 127.19.66.131:58105
I20240502 20:56:36.053756 26980 tablet_copy_service.cc:433] P 297de823852d4a8da44791ab03131cd3: ending tablet copy session c107c43e538d48f58625c4320f0aa4b8-af693f5816e046f592fbd9b2e7ef0ad9 on tablet af693f5816e046f592fbd9b2e7ef0ad9 with peer c107c43e538d48f58625c4320f0aa4b8
W20240502 20:56:36.064214 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:56:36.069494 27227 raft_consensus.cc:1203] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
I20240502 20:56:36.165042 27227 raft_consensus.cc:1203] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14]   Dedup: 1.14->[]
I20240502 20:56:36.232271 14850 ts_tablet_manager.cc:902] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:56:36.232879 14850 tablet_copy_client.cc:308] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:56:36.233753 27248 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet bf189c19164442e5af0827f1fb7dc085 from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:45531)
I20240502 20:56:36.234014 27248 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet bf189c19164442e5af0827f1fb7dc085 from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:45531: session id = 297de823852d4a8da44791ab03131cd3-bf189c19164442e5af0827f1fb7dc085
I20240502 20:56:36.235759 27248 tablet_copy_source_session.cc:205] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:36.237629 14850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf189c19164442e5af0827f1fb7dc085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:36.239903 14850 tablet_copy_client.cc:783] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:36.246904 14850 tablet_copy_client.cc:647] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:56:36.249529 14850 tablet_copy_client.cc:523] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:56:36.251873 14850 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:56:36.264871 27227 raft_consensus.cc:1203] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
I20240502 20:56:36.266638 14850 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:36.267042 14850 tablet_bootstrap.cc:492] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:56:36.267318 14850 ts_tablet_manager.cc:1367] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.016s	user 0.013s	sys 0.003s
I20240502 20:56:36.267400 27277 maintenance_manager.cc:419] P c107c43e538d48f58625c4320f0aa4b8: Scheduling UndoDeltaBlockGCOp(af693f5816e046f592fbd9b2e7ef0ad9): 4840 bytes on disk
I20240502 20:56:36.268100 27162 maintenance_manager.cc:643] P c107c43e538d48f58625c4320f0aa4b8: UndoDeltaBlockGCOp(af693f5816e046f592fbd9b2e7ef0ad9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 20:56:36.268675 14850 raft_consensus.cc:348] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:56:36.269120 14850 raft_consensus.cc:729] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:56:36.269322 14850 consensus_queue.cc:260] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:56:36.270222 14850 ts_tablet_manager.cc:1398] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 20:56:36.271204 27248 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-bf189c19164442e5af0827f1fb7dc085 received from {username='slave'} at 127.19.66.129:45531
I20240502 20:56:36.271350 27248 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 297de823852d4a8da44791ab03131cd3-bf189c19164442e5af0827f1fb7dc085 on tablet bf189c19164442e5af0827f1fb7dc085 with peer 297de823852d4a8da44791ab03131cd3
W20240502 20:56:36.314684 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
I20240502 20:56:36.321321 14850 ts_tablet_manager.cc:902] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3: Initiating tablet copy from peer c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131:44123)
I20240502 20:56:36.321981 14850 tablet_copy_client.cc:308] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.131:44123
I20240502 20:56:36.322764 27248 tablet_copy_service.cc:140] P c107c43e538d48f58625c4320f0aa4b8: Received BeginTabletCopySession request for tablet a906569ab3b14520b4876bc8d5450f7d from peer 297de823852d4a8da44791ab03131cd3 ({username='slave'} at 127.19.66.129:45531)
I20240502 20:56:36.322968 27248 tablet_copy_service.cc:161] P c107c43e538d48f58625c4320f0aa4b8: Beginning new tablet copy session on tablet a906569ab3b14520b4876bc8d5450f7d from peer 297de823852d4a8da44791ab03131cd3 at {username='slave'} at 127.19.66.129:45531: session id = 297de823852d4a8da44791ab03131cd3-a906569ab3b14520b4876bc8d5450f7d
I20240502 20:56:36.324548 27248 tablet_copy_source_session.cc:205] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 20:56:36.325850 14850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a906569ab3b14520b4876bc8d5450f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:56:36.328600 14850 tablet_copy_client.cc:783] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 5 data blocks...
I20240502 20:56:36.336719 14850 tablet_copy_client.cc:647] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3: tablet copy: Starting download of 1 WAL segments...
I20240502 20:56:36.338941 14850 tablet_copy_client.cc:523] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 20:56:36.340555 14850 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3: Bootstrap starting.
I20240502 20:56:36.347460 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(bf189c19164442e5af0827f1fb7dc085): 4925 bytes on disk
I20240502 20:56:36.348011 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(bf189c19164442e5af0827f1fb7dc085) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 20:56:36.352109 14850 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 20:56:36.352713 14850 tablet_bootstrap.cc:492] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3: Bootstrap complete.
I20240502 20:56:36.353188 14850 ts_tablet_manager.cc:1367] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3: Time spent bootstrapping tablet: real 0.013s	user 0.007s	sys 0.007s
I20240502 20:56:36.353927 14850 raft_consensus.cc:348] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:56:36.354091 14850 raft_consensus.cc:729] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297de823852d4a8da44791ab03131cd3, State: Initialized, Role: LEARNER
I20240502 20:56:36.354195 14850 consensus_queue.cc:260] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: NON_VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: true } }
I20240502 20:56:36.354589 14850 ts_tablet_manager.cc:1398] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:56:36.354823 14529 raft_consensus.cc:1050] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER d1f3f8e444364d6faa42d651b5dbb997 to VOTER
I20240502 20:56:36.355433 27248 tablet_copy_service.cc:342] P c107c43e538d48f58625c4320f0aa4b8: Request end of tablet copy session 297de823852d4a8da44791ab03131cd3-a906569ab3b14520b4876bc8d5450f7d received from {username='slave'} at 127.19.66.129:45531
I20240502 20:56:36.355530 27248 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 297de823852d4a8da44791ab03131cd3-a906569ab3b14520b4876bc8d5450f7d on tablet a906569ab3b14520b4876bc8d5450f7d with peer 297de823852d4a8da44791ab03131cd3
I20240502 20:56:36.355437 14529 consensus_queue.cc:237] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } }
I20240502 20:56:36.357565 27090 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:56:36.357892 26960 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:56:36.358245 27227 raft_consensus.cc:1261] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 20:56:36.358423 14863 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:36.358673 14479 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:36.359740 14479 consensus_queue.cc:1035] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 20:56:36.361855 14863 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:36.362473 26960 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:36.363212 27227 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:36.365736 26805 catalog_manager.cc:5484] T 40df15a82a624913b3b712687d999134 P 297de823852d4a8da44791ab03131cd3 reported cstate change: config changed from index 11 to 12, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
I20240502 20:56:36.367563 27090 raft_consensus.cc:2932] T 40df15a82a624913b3b712687d999134 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } }
W20240502 20:56:36.387887 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:56:36.413340 27227 raft_consensus.cc:1203] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
I20240502 20:56:36.413888 14839 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:36.414023 14839 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:56:36.414636 14839 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:56:36.415400 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:56:36.415577 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:56:36.416620 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:56:36.416739 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 20:56:36.416795 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:56:36.416968 14839 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:36.423017 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20240502 20:56:36.487748 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 137 similar messages]
I20240502 20:56:36.501503 14839 raft_consensus.cc:1050] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER c107c43e538d48f58625c4320f0aa4b8 to VOTER
I20240502 20:56:36.502089 14839 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:36.503976 27227 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:36.503976 26960 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:36.504406 27361 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:36.504446 14887 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:36.504855 14839 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240502 20:56:36.505052 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:36.505210 14888 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:36.507249 14887 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.507949 26960 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.508210 27361 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.511510 26807 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 13 to 14, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:36.512804 27227 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.519196 27090 consensus_queue.cc:237] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:36.520823 26960 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 20:56:36.521517 14888 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240502 20:56:36.522361 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:36.522585 27227 raft_consensus.cc:1261] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 20:56:36.523258 14888 consensus_queue.cc:1035] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 20:56:36.523625 14839 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.525967 26960 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.528064 26805 catalog_manager.cc:5484] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 14 to 15, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:56:36.529189 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1b9ff3c921d474fa5ad106d1c17ab00 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:36.530146 27227 raft_consensus.cc:2932] T d1b9ff3c921d474fa5ad106d1c17ab00 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: NON_VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.531471 27342 tablet_service.cc:1514] Processing DeleteTablet for tablet d1b9ff3c921d474fa5ad106d1c17ab00 with delete_type TABLET_DATA_TOMBSTONED (TS ebd8300729ec4226acc38b9cff2d8d3b not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:34124
I20240502 20:56:36.531916 14893 tablet_replica.cc:330] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b: stopping tablet replica
I20240502 20:56:36.532176 14893 raft_consensus.cc:2229] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:56:36.532598 14893 raft_consensus.cc:2258] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:56:36.534210 14893 ts_tablet_manager.cc:1875] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:56:36.538874 14893 ts_tablet_manager.cc:1888] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20240502 20:56:36.538985 14893 log.cc:1196] T d1b9ff3c921d474fa5ad106d1c17ab00 P ebd8300729ec4226acc38b9cff2d8d3b: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-3/wal/wals/d1b9ff3c921d474fa5ad106d1c17ab00
I20240502 20:56:36.539775 26796 catalog_manager.cc:4830] TS ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007): tablet d1b9ff3c921d474fa5ad106d1c17ab00 (table rebalance_test_table_1 [id=2129159ea2e44a4ea9913945352f2626]) successfully deleted
I20240502 20:56:36.600948 27007 maintenance_manager.cc:419] P 297de823852d4a8da44791ab03131cd3: Scheduling UndoDeltaBlockGCOp(a906569ab3b14520b4876bc8d5450f7d): 4586 bytes on disk
I20240502 20:56:36.601601 26895 maintenance_manager.cc:643] P 297de823852d4a8da44791ab03131cd3: UndoDeltaBlockGCOp(a906569ab3b14520b4876bc8d5450f7d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240502 20:56:36.633850 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:56:36.659961 14863 raft_consensus.cc:1050] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b: attempting to promote NON_VOTER c107c43e538d48f58625c4320f0aa4b8 to VOTER
I20240502 20:56:36.660465 14863 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:36.662130 27227 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:36.662146 27090 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:36.662813 14863 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:36.663046 26960 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:36.663070 14528 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:36.663569 14863 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:36.665722 14479 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.666743 26960 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.668083 27090 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.669636 26807 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 12 to 13, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:36.671147 27227 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.675623 27361 consensus_queue.cc:237] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:36.677637 26960 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:36.677631 27227 raft_consensus.cc:1261] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer ebd8300729ec4226acc38b9cff2d8d3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:36.678164 14863 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:36.679057 14863 consensus_queue.cc:1035] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:36.680183 14528 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.681813 26960 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.683058 26796 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5586b9e60b544aed87c24f7b3ce0198c with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:36.684814 26807 catalog_manager.cc:5484] T 5586b9e60b544aed87c24f7b3ce0198c P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 13 to 14, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 20:56:36.686282 27227 raft_consensus.cc:2932] T 5586b9e60b544aed87c24f7b3ce0198c P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.689652 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet 5586b9e60b544aed87c24f7b3ce0198c with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:47758
I20240502 20:56:36.690219 14912 tablet_replica.cc:330] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:56:36.690467 14912 raft_consensus.cc:2229] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:56:36.690848 14912 raft_consensus.cc:2258] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:56:36.691744 14912 ts_tablet_manager.cc:1875] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:56:36.695950 14912 ts_tablet_manager.cc:1888] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20240502 20:56:36.696133 14912 log.cc:1196] T 5586b9e60b544aed87c24f7b3ce0198c P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/5586b9e60b544aed87c24f7b3ce0198c
I20240502 20:56:36.696832 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet 5586b9e60b544aed87c24f7b3ce0198c (table rebalance_test_table_3 [id=aab016c806224a07b904d99b93e2e5b9]) successfully deleted
W20240502 20:56:36.700114 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20240502 20:56:36.719821 26960 raft_consensus.cc:1203] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16]   Dedup: 1.16->[]
I20240502 20:56:36.719821 26959 raft_consensus.cc:1203] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
I20240502 20:56:36.730700 14839 raft_consensus.cc:1050] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997: attempting to promote NON_VOTER c107c43e538d48f58625c4320f0aa4b8 to VOTER
I20240502 20:56:36.731200 14839 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:36.732924 26959 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 20:56:36.732985 27227 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 20:56:36.732990 27361 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 20:56:36.733525 14833 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 20:56:36.733727 14888 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 20:56:36.733916 14887 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 20:56:36.737349 14888 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.737864 27227 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.737892 26959 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.739997 27361 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.741250 26807 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 14 to 15, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:36.744809 27090 consensus_queue.cc:237] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:36.750265 27227 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240502 20:56:36.750876 27361 raft_consensus.cc:1261] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240502 20:56:36.750808 14833 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 20:56:36.751364 14888 consensus_queue.cc:1035] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 20:56:36.753415 14888 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.755551 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f00c4b7954564820875a25da5834416a with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:36.755769 27227 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.756809 26802 catalog_manager.cc:5484] T f00c4b7954564820875a25da5834416a P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 15 to 16, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New cstate: current_term: 1 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:36.760363 27361 raft_consensus.cc:2932] T f00c4b7954564820875a25da5834416a P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER 297de823852d4a8da44791ab03131cd3 (127.19.66.129) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.760903 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet f00c4b7954564820875a25da5834416a with delete_type TABLET_DATA_TOMBSTONED (TS 297de823852d4a8da44791ab03131cd3 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:35532
I20240502 20:56:36.761454 14923 tablet_replica.cc:330] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3: stopping tablet replica
I20240502 20:56:36.761626 14923 raft_consensus.cc:2229] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:56:36.761802 14923 raft_consensus.cc:2258] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:56:36.763536 14923 ts_tablet_manager.cc:1875] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 20:56:36.764485 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:56:36.768934 14923 ts_tablet_manager.cc:1888] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.15
I20240502 20:56:36.769032 14923 log.cc:1196] T f00c4b7954564820875a25da5834416a P 297de823852d4a8da44791ab03131cd3: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-0/wal/wals/f00c4b7954564820875a25da5834416a
I20240502 20:56:36.769726 26794 catalog_manager.cc:4830] TS 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835): tablet f00c4b7954564820875a25da5834416a (table rebalance_test_table_2 [id=fde883c6e2f44181aad0a49a138415ab]) successfully deleted
W20240502 20:56:36.810580 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cd1b1316da2d47938164ac5b75f11bae: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:36.818329 26799 maintenance_manager.cc:419] P e0c668105a124f0eb7227390002cc15e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3374 bytes on disk
I20240502 20:56:36.821544 26798 maintenance_manager.cc:643] P e0c668105a124f0eb7227390002cc15e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s	user 0.003s	sys 0.000s Metrics: {"cfile_init":1,"dirs.queue_time_us":278,"dirs.run_cpu_time_us":75,"dirs.run_wall_time_us":76,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"thread_start_us":170,"threads_started":1}
I20240502 20:56:36.858237 14884 raft_consensus.cc:1050] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3: attempting to promote NON_VOTER c107c43e538d48f58625c4320f0aa4b8 to VOTER
I20240502 20:56:36.858777 14884 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:36.860469 27361 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:36.860469 27227 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEARNER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:36.860981 27090 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 20:56:36.861022 14941 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:36.861522 14772 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:36.862277 14772 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 20:56:36.864590 14883 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.865545 27090 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.865590 27361 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.867401 27227 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.868781 26806 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b reported cstate change: config changed from index 12 to 13, c107c43e538d48f58625c4320f0aa4b8 (127.19.66.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.873107 26959 consensus_queue.cc:237] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } }
I20240502 20:56:36.874614 27227 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:36.874642 27361 raft_consensus.cc:1261] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer 297de823852d4a8da44791ab03131cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:36.875141 14884 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:36.875443 14883 consensus_queue.cc:1035] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:36.877393 14884 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P 297de823852d4a8da44791ab03131cd3 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.877849 27227 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.877974 27361 raft_consensus.cc:2932] T af693f5816e046f592fbd9b2e7ef0ad9 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.879257 26794 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af693f5816e046f592fbd9b2e7ef0ad9 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:36.881248 26805 catalog_manager.cc:5484] T af693f5816e046f592fbd9b2e7ef0ad9 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 13 to 14, VOTER d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130) evicted. New cstate: current_term: 1 leader_uuid: "297de823852d4a8da44791ab03131cd3" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { promote: false } } }
I20240502 20:56:36.885233 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet af693f5816e046f592fbd9b2e7ef0ad9 with delete_type TABLET_DATA_TOMBSTONED (TS d1f3f8e444364d6faa42d651b5dbb997 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:47758
I20240502 20:56:36.885530 14912 tablet_replica.cc:330] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997: stopping tablet replica
I20240502 20:56:36.885668 14912 raft_consensus.cc:2229] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 20:56:36.885977 14912 raft_consensus.cc:2258] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 20:56:36.886816 14912 ts_tablet_manager.cc:1875] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 20:56:36.890455 14912 ts_tablet_manager.cc:1888] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20240502 20:56:36.890558 14912 log.cc:1196] T af693f5816e046f592fbd9b2e7ef0ad9 P d1f3f8e444364d6faa42d651b5dbb997: Deleting WAL directory at /tmp/dist-test-taskF51BWD/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714682727169490-19722-0/raft_consensus-itest-cluster/ts-1/wal/wals/af693f5816e046f592fbd9b2e7ef0ad9
I20240502 20:56:36.891199 26797 catalog_manager.cc:4830] TS d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307): tablet af693f5816e046f592fbd9b2e7ef0ad9 (table rebalance_test_table_9 [id=1b230b3b28de45dcade302c6a436ebad]) successfully deleted
W20240502 20:56:36.929304 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20240502 20:56:36.951686 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20240502 20:56:36.952338 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20240502 20:56:37.011660 14833 consensus_queue.cc:579] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Leader has been unable to successfully communicate with peer 1f40a11333e44ea0a34b6bc7618af522 for more than 300 seconds (300.287s)
I20240502 20:56:37.015527 27090 consensus_queue.cc:237] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 11, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } }
I20240502 20:56:37.018373 27361 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 3 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240502 20:56:37.018723 27227 raft_consensus.cc:1261] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 3 FOLLOWER]: Refusing update from remote peer d1f3f8e444364d6faa42d651b5dbb997: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240502 20:56:37.019111 14682 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 20:56:37.019313 14833 consensus_queue.cc:1035] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240502 20:56:37.019502 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 20:56:37.019496 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 20:56:37.021737 14682 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:56:37.022403 27361 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P ebd8300729ec4226acc38b9cff2d8d3b [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 14 to 15, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:56:37.025439 26797 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a547561343304137a044ce11517f6007 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 20:56:37.025895 26807 catalog_manager.cc:5484] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 14 to 15, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New cstate: current_term: 3 leader_uuid: "d1f3f8e444364d6faa42d651b5dbb997" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:37.025933 27227 raft_consensus.cc:2932] T a547561343304137a044ce11517f6007 P c107c43e538d48f58625c4320f0aa4b8 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 14 to 15, NON_VOTER 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: NON_VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { promote: false } } }
I20240502 20:56:37.238301 14563 raft_consensus.cc:1050] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:56:37.238817 14563 consensus_queue.cc:237] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:56:37.240689 27090 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:37.240658 26959 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:37.240716 27361 raft_consensus.cc:1261] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 20:56:37.241325 14960 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:37.241536 14563 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:37.241919 14882 consensus_queue.cc:1035] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 20:56:37.245008 14882 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:37.246484 27091 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:37.246608 26959 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:37.249198 27361 raft_consensus.cc:2932] T a906569ab3b14520b4876bc8d5450f7d P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:37.251646 26802 catalog_manager.cc:5484] T a906569ab3b14520b4876bc8d5450f7d P d1f3f8e444364d6faa42d651b5dbb997 reported cstate change: config changed from index 13 to 14, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
W20240502 20:56:37.287612 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20240502 20:56:37.315212 14961 raft_consensus.cc:1050] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8: attempting to promote NON_VOTER 297de823852d4a8da44791ab03131cd3 to VOTER
I20240502 20:56:37.315831 14961 consensus_queue.cc:237] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:56:37.317441 27091 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240502 20:56:37.317490 26959 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3 [term 1 LEARNER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240502 20:56:37.317704 27361 raft_consensus.cc:1261] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Refusing update from remote peer c107c43e538d48f58625c4320f0aa4b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240502 20:56:37.318126 14563 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240502 20:56:37.318429 14960 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240502 20:56:37.318612 14882 consensus_queue.cc:1035] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240502 20:56:37.321496 14882 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:37.322150 26959 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P 297de823852d4a8da44791ab03131cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:37.324179 27361 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
I20240502 20:56:37.324823 26806 catalog_manager.cc:5484] T bf189c19164442e5af0827f1fb7dc085 P c107c43e538d48f58625c4320f0aa4b8 reported cstate change: config changed from index 16 to 17, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c107c43e538d48f58625c4320f0aa4b8" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 20:56:37.325647 27091 raft_consensus.cc:2932] T bf189c19164442e5af0827f1fb7dc085 P d1f3f8e444364d6faa42d651b5dbb997 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 297de823852d4a8da44791ab03131cd3 (127.19.66.129) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c107c43e538d48f58625c4320f0aa4b8" member_type: VOTER last_known_addr { host: "127.19.66.131" port: 44123 } attrs { replace: true } } peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } } }
W20240502 20:56:37.383111 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20240502 20:56:37.446906 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 20:56:37.482510 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:56:37.552187 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20240502 20:56:37.898953 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:38.120790 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20240502 20:56:38.193838 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20240502 20:56:38.305246 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20240502 20:56:38.542392 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240502 20:56:38.543430 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:56:38.796602 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:56:38.861032 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240502 20:56:38.991165 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20240502 20:56:39.000491 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 20:56:39.062959 14528 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:39.063207 14528 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:56:39.064006 14528 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:56:39.064910 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:56:39.065238 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:56:39.065845 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:39.065974 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:39.066030 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:56:39.066336 14528 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:39.068750 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:56:39.096064 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:39.116072 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20240502 20:56:39.298888 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20240502 20:56:39.355819 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:56:39.419289 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20240502 20:56:39.462519 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20240502 20:56:39.598496 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20240502 20:56:39.659945 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:56:39.733019 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
I20240502 20:56:39.743216 14528 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:39.743424 14528 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:56:39.744092 14528 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:56:39.744948 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:56:39.745189 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:56:39.745764 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:56:39.746079 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:56:39.746160 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:56:39.746518 14528 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:39.847007 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 20:56:39.952952 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20240502 20:56:40.006397 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20240502 20:56:40.119863 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:56:40.136152 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f1545142b0bd40859bf33bf8686002ae: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:56:40.266621 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 20:56:40.390894 14528 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:56:40.391064 14528 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:56:40.391748 14528 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:56:40.392503 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:56:40.392534 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:56:40.393021 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:56:40.393216 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:56:40.393299 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 1 yes votes; 4 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:56:40.393474 14528 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:56:40.518599 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20240502 20:56:40.664038 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20240502 20:56:40.730188 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:40.761270 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:56:41.186405 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:56:41.316246 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:41.382997 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:56:41.453375 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:56:41.529635 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 113 similar messages]
I20240502 20:56:41.610185 14883 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:41.610368 14883 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:56:41.611043 14883 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:56:41.611475 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:56:41.611636 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:56:41.612417 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:56:41.612509 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:41.612600 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:56:41.612751 14883 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:41.640213 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20240502 20:56:41.706480 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
I20240502 20:56:41.873242 27031 tablet_copy_service.cc:454] P d1f3f8e444364d6faa42d651b5dbb997: tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-d1b9ff3c921d474fa5ad106d1c17ab00 on tablet d1b9ff3c921d474fa5ad106d1c17ab00 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 has expired after 609.480s of idle time
I20240502 20:56:41.873409 27031 tablet_copy_service.cc:433] P d1f3f8e444364d6faa42d651b5dbb997: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-d1b9ff3c921d474fa5ad106d1c17ab00 on tablet d1b9ff3c921d474fa5ad106d1c17ab00 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:56:41.929908 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:56:41.949160 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20240502 20:56:41.954221 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:56:41.962806 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:56:42.037402 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:42.055750 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20240502 20:56:42.079696 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
I20240502 20:56:42.092289 27166 tablet_copy_service.cc:454] P c107c43e538d48f58625c4320f0aa4b8: tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-a547561343304137a044ce11517f6007 on tablet a547561343304137a044ce11517f6007 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3 has expired after 609.805s of idle time
I20240502 20:56:42.092445 27166 tablet_copy_service.cc:433] P c107c43e538d48f58625c4320f0aa4b8: ending tablet copy session 1af1e87d6ff2448d9502de9d7c2b8ef3-a547561343304137a044ce11517f6007 on tablet a547561343304137a044ce11517f6007 with peer 1af1e87d6ff2448d9502de9d7c2b8ef3
W20240502 20:56:42.359143 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 20:56:42.384613 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20240502 20:56:42.473253 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20240502 20:56:42.503906 26796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:56:42.504876 26796 catalog_manager.cc:4573] TS 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5cf737e20ca74db1b94ccff5d60133ee: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:56:42.803387 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:56:42.987195 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:43.313773 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:43.315940 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:56:43.336385 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
I20240502 20:56:43.468894 15285 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:43.469153 15285 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:56:43.469744 15285 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:56:43.470414 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:56:43.470603 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:56:43.471174 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:56:43.471339 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:43.471429 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:56:43.471585 15285 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:43.601007 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240502 20:56:43.601796 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:56:43.603955 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00c4b7954564820875a25da5834416a: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:43.862085 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:56:43.940266 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:56:44.032667 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:44.112555 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20240502 20:56:44.191901 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:44.327369 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20240502 20:56:44.339648 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:44.350462 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:56:44.353781 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:56:44.369421 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:44.378513 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:56:44.608626 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:44.692390 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20240502 20:56:44.729753 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 20:56:45.222776 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:45.266979 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:45.367292 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:56:45.426388 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a547561343304137a044ce11517f6007: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:45.476250 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 20:56:45.755968 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:56:45.830904 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:45.898546 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:46.060838 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:56:46.309020 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:56:46.480942 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:46.502904 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:56:46.535005 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 114 similar messages]
W20240502 20:56:46.536065 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:46.604377 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:46.685400 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:46.713157 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:46.753883 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:56:46.783583 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:46.880681 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:56:46.944031 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20240502 20:56:47.004575 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:47.010805 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 20:56:47.672935 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:48.083487 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:56:48.087352 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:48.330750 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
I20240502 20:56:48.332051 15479 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:48.332216 15479 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:56:48.332741 15479 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:56:48.333292 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:56:48.333506 27358 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:56:48.334169 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:48.334266 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:48.334304 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:56:48.334432 15479 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:48.367240 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:48.441790 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:56:48.590529 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:56:48.636548 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:56:48.697943 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:56:48.844178 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:49.040328 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:49.086234 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:56:49.102699 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:49.145185 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:56:49.145612 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 20:56:49.146099 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cd1b1316da2d47938164ac5b75f11bae: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:56:49.166003 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:49.240299 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20240502 20:56:49.257442 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:49.261147 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:49.369612 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:56:49.374495 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:56:49.496119 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:49.624840 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 20:56:49.639931 14528 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:56:49.640157 14528 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:56:49.641569 14528 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:56:49.641541 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:56:49.641907 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:56:49.642745 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:49.642844 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:56:49.643033 14528 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:56:49.643277 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:56:50.236627 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:50.436614 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:50.606460 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:56:50.711433 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:50.732836 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:56:50.738180 26796 catalog_manager.cc:4573] TS 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5cf737e20ca74db1b94ccff5d60133ee: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:56:50.898680 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:56:51.115252 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:56:51.290558 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:51.306062 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:51.383620 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:56:51.438364 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:51.518124 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:51.604641 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:56:51.677150 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240502 20:56:51.677942 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:51.728039 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:51.749897 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:51.749850 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20240502 20:56:51.834123 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00c4b7954564820875a25da5834416a: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:51.869689 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 20:56:51.921046 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:56:52.183997 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 20:56:52.222285 14883 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:52.222525 14883 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:56:52.223327 14883 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:56:52.224006 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:56:52.224277 27358 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:56:52.224685 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:56:52.224787 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:52.224856 26893 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:56:52.225028 14883 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:52.258702 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:52.327108 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
I20240502 20:56:52.851136 15479 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:52.851361 15479 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:56:52.852150 15479 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:56:52.852947 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:56:52.853262 27358 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:56:52.853680 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:52.853818 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:52.853900 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:56:52.854120 15479 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:53.043865 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:53.127388 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:56:53.180294 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:53.487251 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:56:53.494212 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:56:53.551040 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:53.624996 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a547561343304137a044ce11517f6007: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:56:53.753551 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:56:53.823715 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:53.901271 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:56:53.928134 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:56:54.011940 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:54.090402 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:56:54.091619 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:54.109176 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:54.114784 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:54.313747 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:56:54.327064 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20240502 20:56:54.328960 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:54.608646 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:56:54.614017 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:54.655544 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 20:56:54.917569 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
I20240502 20:56:55.384008 15873 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:56:55.384189 15873 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:56:55.384769 15873 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:56:55.386148 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:56:55.386818 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:55.386898 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:56:55.387147 15873 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:55.478260 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:56:55.625653 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:56:55.688616 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:55.841416 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 20:56:56.052706 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:56:56.096621 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:56.187309 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:56:56.239982 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:56.267658 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:56:56.273195 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:56:56.353941 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:56.391175 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 20:56:56.435235 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:56:56.547820 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:56.701962 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:56.782979 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240502 20:56:56.810017 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20240502 20:56:56.919871 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:56.931479 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 20:56:56.997157 15931 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:56:56.997373 15931 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:56:56.997938 15931 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:56:56.998548 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:56:56.998733 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:56:56.999296 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:56:56.999468 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:56:56.999542 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:56:56.999725 15931 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:56:57.105983 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:56:57.234990 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:56:57.518612 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:56:57.974668 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:58.092453 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:56:58.180229 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:56:58.327253 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:56:58.457207 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:58.783577 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:56:58.826745 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:56:58.836061 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:56:58.886420 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:56:58.898056 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:56:58.980362 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:56:59.133698 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:59.171566 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20240502 20:56:59.275038 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:56:59.289134 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:56:59.371008 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20240502 20:56:59.383072 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:56:59.462090 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:56:59.495187 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:56:59.588620 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 20:56:59.911660 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:57:00.152732 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:57:00.477972 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:57:00.738798 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:00.806548 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:57:00.938416 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:57:00.940114 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:57:00.941006 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:57:01.160782 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:57:01.191100 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:57:01.289295 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:01.394707 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:57:01.436478 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:57:01.641806 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:57:01.774667 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:01.888360 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240502 20:57:01.948216 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:57:01.973680 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:57:01.992151 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:57:02.011418 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:57:02.032102 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20240502 20:57:02.122411 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 20:57:02.397409 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:57:02.517329 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:02.907685 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:57:02.998116 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:03.250814 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:57:03.274689 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:57:03.574307 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:57:03.787055 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:57:03.867772 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:57:03.904919 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:03.992929 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:57:03.993796 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:04.011855 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:04.108052 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:57:04.254763 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:04.366914 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:04.391676 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:57:04.421412 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:57:04.478045 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:57:04.489395 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 20:57:04.551312 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20240502 20:57:04.630043 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:04.887408 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:05.009001 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:57:05.461237 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:05.478473 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
I20240502 20:57:05.495007 16419 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:05.495224 16419 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:57:05.495963 16419 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:57:05.496670 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:05.496917 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:57:05.497679 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:05.497818 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:05.497879 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:57:05.498044 16419 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:05.561390 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:57:05.562582 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet cd1b1316da2d47938164ac5b75f11bae: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:57:05.684310 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:57:05.960124 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:57:06.012689 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:57:06.145530 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:57:06.222594 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:06.320616 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:57:06.382467 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:57:06.448330 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:57:06.494352 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20240502 20:57:06.553560 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20240502 20:57:06.737515 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:06.773389 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:06.777158 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:06.813094 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 20:57:06.817410 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20240502 20:57:06.841259 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
I20240502 20:57:06.902067 16485 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:06.902319 16485 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:57:06.903043 16485 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:57:06.903604 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:06.903836 27358 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:57:06.904623 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:57:06.904760 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:06.904852 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:57:06.905113 16485 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:07.033867 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240502 20:57:07.064105 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:07.174646 26796 catalog_manager.cc:4573] TS 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5cf737e20ca74db1b94ccff5d60133ee: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:57:07.213099 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20240502 20:57:07.338600 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 20:57:07.529210 16498 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:57:07.529444 16498 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:57:07.530090 16498 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:57:07.531343 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:57:07.531487 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:57:07.531617 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:57:07.531785 16498 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:08.044942 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:08.166031 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:57:08.250247 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00c4b7954564820875a25da5834416a: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:08.306648 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:08.603251 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:57:08.608865 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:57:08.617292 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:57:08.646518 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:08.791927 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:57:09.000356 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:57:09.015179 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20240502 20:57:09.214277 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:09.223965 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20240502 20:57:09.240077 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:09.243356 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:09.277667 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20240502 20:57:09.364351 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 20:57:09.484333 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 20:57:09.543558 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:57:09.544683 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
I20240502 20:57:09.580272 16616 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:57:09.580503 16616 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:57:09.581434 16616 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:57:09.581835 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:57:09.582778 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:57:09.582933 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:57:09.583132 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:09.583249 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:57:09.583552 16616 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:57:09.721217 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20240502 20:57:09.876528 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:57:09.935281 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:57:10.032169 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a547561343304137a044ce11517f6007: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:10.039343 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:10.483613 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20240502 20:57:10.730490 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:57:10.827373 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20240502 20:57:10.950789 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:57:11.094919 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:57:11.214133 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:57:11.325713 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20240502 20:57:11.394052 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20240502 20:57:11.398404 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:57:11.506691 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 20:57:11.564515 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:57:11.609819 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20240502 20:57:11.621919 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:11.676844 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
I20240502 20:57:11.796139 16618 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:11.796351 16618 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:57:11.797149 16618 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:57:11.797610 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:11.797854 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:57:11.798811 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:11.798964 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:11.799012 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:57:11.799245 16618 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:11.806663 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20240502 20:57:11.807067 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:11.978765 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:11.982762 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 20:57:12.059221 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240502 20:57:12.153574 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:12.240013 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20240502 20:57:12.663251 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:57:12.984591 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 20:57:13.001152 16811 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:13.001374 16811 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:57:13.002028 16811 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:57:13.002645 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:57:13.002883 26959 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:57:13.003331 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:57:13.003512 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:57:13.003598 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:57:13.003749 16811 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:13.011267 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:13.268632 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:13.479053 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:57:13.554831 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:57:13.758533 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:13.781050 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:57:13.806380 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:57:13.848944 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:13.997295 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20240502 20:57:14.040189 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:57:14.092142 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:14.244483 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:14.283164 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20240502 20:57:14.286113 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 20:57:14.408443 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20240502 20:57:14.461817 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:14.560138 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:57:14.561041 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
I20240502 20:57:14.579078 16846 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:14.579491 16846 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:57:14.580233 16846 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:57:14.580796 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:57:14.580982 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:57:14.580987 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:57:14.581719 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:14.581835 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:14.581897 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:57:14.582151 16846 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:14.772450 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20240502 20:57:15.215404 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:57:15.457715 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:57:15.515650 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20240502 20:57:15.806202 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:57:15.847244 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20240502 20:57:16.032498 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:57:16.189805 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20240502 20:57:16.198693 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20240502 20:57:16.354219 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:57:16.364790 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:57:16.523404 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:57:16.609318 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
I20240502 20:57:16.625821 16846 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:57:16.626034 16846 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:57:16.626777 16846 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:57:16.627235 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:57:16.627701 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:57:16.628274 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:16.628472 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:57:16.628700 16846 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:57:16.629148 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:57:16.654224 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:16.700503 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:16.723306 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 20:57:16.842855 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20240502 20:57:16.886279 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:57:16.903132 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:16.909844 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20240502 20:57:17.053977 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20240502 20:57:17.079470 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240502 20:57:17.211508 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:17.456558 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:57:17.896893 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:57:17.959452 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:17.961786 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:57:18.356863 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
I20240502 20:57:18.519407 17026 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:57:18.519623 17026 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:57:18.520382 17026 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:57:18.520974 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:57:18.522401 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:57:18.522513 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:18.522603 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:57:18.522789 17026 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 20:57:18.522827 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:57:18.533849 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:57:18.730048 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:57:18.750793 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:57:18.885942 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:18.932617 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:18.987596 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 20:57:19.011801 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
I20240502 20:57:19.015666 17085 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:19.015838 17085 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
W20240502 20:57:19.016268 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 20:57:19.016391 17085 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:57:19.016808 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:19.016974 27358 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:57:19.018309 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:19.018595 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:19.018693 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:57:19.018915 17085 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:19.094066 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:19.117728 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20240502 20:57:19.336885 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20240502 20:57:19.403735 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:19.587939 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:19.626370 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:57:19.627287 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20240502 20:57:19.646440 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240502 20:57:19.653000 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20240502 20:57:19.854339 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:57:20.131285 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20240502 20:57:20.346007 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:57:20.483454 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:57:20.533480 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
I20240502 20:57:20.712839 17085 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:20.713140 17085 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:57:20.713879 17085 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:57:20.714567 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:20.714787 27358 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:57:20.715889 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:20.716090 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:20.716192 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:57:20.716392 17085 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:57:20.803267 17221 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:57:20.803532 17221 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:57:20.804350 17221 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:57:20.805012 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:57:20.805521 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:57:20.806398 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:20.806514 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:57:20.806612 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:57:20.806715 17221 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:57:21.109230 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:57:21.117210 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:57:21.280871 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:57:21.486439 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:21.517891 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20240502 20:57:21.542313 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:57:21.594388 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20240502 20:57:21.664014 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20240502 20:57:21.697933 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 20:57:21.729018 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:21.801283 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20240502 20:57:21.903792 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:57:21.967661 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20240502 20:57:22.042788 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:22.093322 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240502 20:57:22.242549 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
I20240502 20:57:22.260202 17298 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:22.260455 17298 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:57:22.261240 17298 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:57:22.261993 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:57:22.262223 26959 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:57:22.262763 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:57:22.263077 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:22.263167 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:57:22.263520 17298 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:22.284821 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:22.315604 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:22.764461 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:57:22.930210 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:57:23.010864 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:23.198014 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 20:57:23.378198 17343 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:57:23.378444 17343 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:57:23.379083 17343 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:57:23.380638 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:57:23.380806 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:57:23.380950 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:57:23.381187 17343 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:23.529544 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:57:23.547089 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:57:23.688701 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:57:23.734997 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:23.767350 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20240502 20:57:24.079370 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:57:24.097019 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 20:57:24.184818 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:24.217389 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20240502 20:57:24.248879 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:24.301581 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:24.461589 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20240502 20:57:24.626446 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:24.662146 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:57:24.662838 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20240502 20:57:24.691990 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20240502 20:57:24.754964 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240502 20:57:24.773777 17390 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:24.774011 17390 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:57:24.774709 17390 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:57:24.775344 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:24.775557 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:57:24.776196 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:24.776325 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:24.776384 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:57:24.776562 17390 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:24.809638 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20240502 20:57:25.284452 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:57:25.560667 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:57:25.576135 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:57:25.704502 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:26.059826 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:57:26.063256 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:57:26.112219 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:57:26.130388 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20240502 20:57:26.482777 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:26.555758 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:26.607625 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20240502 20:57:26.674985 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:57:26.711652 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 20:57:26.791632 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20240502 20:57:26.792160 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:27.093677 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:27.112720 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240502 20:57:27.122100 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:27.128512 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:27.226940 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:57:27.320490 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:27.445899 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:27.958156 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:57:28.003566 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:28.169677 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:57:28.230525 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:57:28.295337 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:57:28.531508 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
I20240502 20:57:28.534099 17244 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:28.534299 17244 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:57:28.535207 17244 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:57:28.536007 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:28.536283 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:57:28.536829 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:57:28.537283 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:57:28.537364 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:57:28.537698 17244 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:28.662776 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:57:28.706938 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:57:28.768895 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:28.993042 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:57:29.081211 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 20:57:29.119240 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:29.138906 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:29.202947 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:29.215919 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:29.226480 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20240502 20:57:29.606649 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:29.643952 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:29.738879 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:57:29.749133 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:29.765547 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:57:29.888744 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
I20240502 20:57:30.159822 17244 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:30.160058 17244 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:57:30.160699 17244 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:57:30.161392 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:30.161607 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:57:30.162256 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:57:30.162408 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:57:30.162482 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:57:30.162659 17244 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:30.451695 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:30.543051 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:57:30.619724 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:57:30.689563 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:57:31.010550 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:57:31.022979 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:31.141295 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:57:31.241171 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:57:31.277876 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:31.429706 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:57:31.488638 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 20:57:31.538864 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:31.636960 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:31.677778 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20240502 20:57:31.767906 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
I20240502 20:57:31.773811 17719 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:57:31.774006 17719 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:57:31.774801 17719 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:57:31.775187 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:57:31.775897 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:57:31.776361 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:57:31.776494 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:57:31.776548 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:57:31.776741 17719 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:57:31.844584 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:32.068578 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:32.118126 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240502 20:57:32.169353 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:32.273583 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:32.345021 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:32.551952 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240502 20:57:33.013015 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:57:33.014205 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:57:33.095778 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:33.274247 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:57:33.393854 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:57:33.527987 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:33.557209 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:33.653131 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:57:33.744760 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:57:33.751580 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:57:33.982791 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:34.019373 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 20:57:34.071305 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:34.119798 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:34.247834 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:34.270615 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20240502 20:57:34.526502 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:34.725342 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:34.752655 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:57:34.756366 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:34.823029 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:34.863059 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:57:35.208000 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20240502 20:57:35.395033 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:57:35.471101 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:57:35.526216 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:57:35.995187 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:36.150148 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:57:36.158778 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:57:36.234826 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:36.321835 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:57:36.399590 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:57:36.403708 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20240502 20:57:36.492466 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 20:57:36.541127 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:36.562852 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:36.680491 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:36.728029 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20240502 20:57:36.739974 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20240502 20:57:37.063257 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20240502 20:57:37.135448 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:37.174142 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 114 similar messages]
W20240502 20:57:37.442777 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:37.656282 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:57:37.914191 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:57:37.915403 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:37.976326 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:38.018455 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:57:38.055253 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:57:38.234324 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:57:38.419376 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
I20240502 20:57:38.436899 18097 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:38.437250 18097 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:57:38.438107 18097 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
W20240502 20:57:38.439427 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:57:38.439419 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:57:38.439631 26959 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:57:38.439934 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:38.440011 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:57:38.440158 18097 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:38.517426 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:38.672820 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:57:38.785923 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:57:38.847715 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:57:38.929548 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:38.988839 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:57:39.055142 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 20:57:39.105167 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:39.133414 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:39.176299 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20240502 20:57:39.222570 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20240502 20:57:39.250078 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:39.405355 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20240502 20:57:39.425402 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20240502 20:57:39.702721 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:39.757159 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:57:40.016736 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 20:57:40.141144 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f1545142b0bd40859bf33bf8686002ae: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:57:40.143726 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
I20240502 20:57:40.260018 18169 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:57:40.260169 18169 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:57:40.260638 18169 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:57:40.262032 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:57:40.262138 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:57:40.262225 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:57:40.262398 18169 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:40.367661 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:40.542451 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:57:40.580652 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:57:40.708019 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:57:40.826901 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:40.957971 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 20:57:41.181128 18039 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:41.181311 18039 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:57:41.181974 18039 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:57:41.182478 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:41.182647 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:57:41.183543 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:41.183714 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:41.183750 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:57:41.184024 18039 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:41.199518 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20240502 20:57:41.211706 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:57:41.212453 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f2dfc0bbbbca4cc28a97480007343424: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:57:41.373412 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:57:41.389930 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20240502 20:57:41.511583 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 20:57:41.586021 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:41.621623 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:41.662197 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:41.663776 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:41.720669 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 20:57:41.766613 18318 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:41.766829 18318 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:57:41.767418 18318 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:57:41.768024 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:57:41.768314 26959 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:57:41.768733 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:57:41.769166 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:41.769275 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:57:41.769445 18318 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:42.022023 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:42.110759 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20240502 20:57:42.185243 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240502 20:57:42.185957 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:42.825610 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:57:42.843216 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a547561343304137a044ce11517f6007: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:42.875399 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:42.928992 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:57:43.005961 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:43.236433 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:57:43.239468 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:43.317718 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:57:43.406031 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:57:43.627480 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:57:43.730633 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:43.862120 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:43.884245 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:57:43.980057 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:44.015738 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:57:44.136400 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:44.157608 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:44.219971 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:44.220638 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 20:57:44.555697 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:44.724121 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20240502 20:57:44.822355 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:45.069854 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:57:45.270540 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20240502 20:57:45.271812 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:57:45.294358 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:45.472347 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:45.694666 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:45.735872 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:57:45.761448 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:57:45.847561 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:57:46.073597 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:57:46.074090 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:46.358927 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:57:46.424839 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:46.477145 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:57:46.594064 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:46.653606 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:46.719293 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 20:57:46.764621 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:46.851380 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:47.172212 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20240502 20:57:47.174329 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:47.238579 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 20:57:47.291399 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:47.819589 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:57:47.837600 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:47.853042 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:57:47.864593 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:48.053647 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
I20240502 20:57:48.062551 18466 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:57:48.062743 18466 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:57:48.064031 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:57:48.064404 18466 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
W20240502 20:57:48.064908 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:48.065323 26959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:57:48.065866 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:57:48.066051 18466 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:57:48.066145 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:57:48.234678 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:57:48.280052 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:57:48.441812 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:48.466746 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:57:48.601107 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:57:48.891753 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20240502 20:57:48.925437 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:57:48.962373 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:49.122910 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:57:49.157819 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:49.295279 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:49.458632 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 20:57:49.461387 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:49.471184 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:49.701930 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
I20240502 20:57:49.814256 18695 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:49.814445 18695 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:57:49.815053 18695 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:57:49.815898 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:49.816221 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:57:49.816805 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:57:49.816995 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:49.817057 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:57:49.817287 18695 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:49.893563 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:50.287676 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:57:50.344257 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:57:50.358748 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:50.445745 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:50.526505 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:57:50.533362 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:57:50.592442 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:50.812999 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:50.924588 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:57:50.967314 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:57:51.302098 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:57:51.382014 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:57:51.405548 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:51.445463 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:51.465358 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:57:51.672660 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:51.731281 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:51.806206 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:51.898941 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 20:57:51.903446 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
I20240502 20:57:52.081852 18831 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:52.082087 18831 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:57:52.082952 18831 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:57:52.083428 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:57:52.083640 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:57:52.084293 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:57:52.084491 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:52.084566 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:57:52.084702 18831 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:52.239470 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 20:57:52.266804 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20240502 20:57:52.299139 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:52.698201 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:52.856382 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:57:52.991638 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:57:53.008621 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 20:57:53.090734 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:53.295222 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:53.326313 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:57:53.485595 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:57:53.502447 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:53.740082 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:53.743439 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:57:53.893390 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:57:53.902690 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:54.087160 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:57:54.125452 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:54.298457 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:54.454176 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:54.496516 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 20:57:54.655995 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:54.701265 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:54.780979 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20240502 20:57:55.303728 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:55.360178 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:57:55.380502 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:57:55.405500 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:57:55.427309 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:57:55.640897 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:57:55.665107 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:57:55.775671 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20240502 20:57:55.885519 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:56.032933 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:56.223368 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:57:56.299808 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:57:56.414707 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:56.525002 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20240502 20:57:56.648586 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:56.843211 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:56.883437 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:57:56.902508 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 20:57:56.926425 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:57.089008 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20240502 20:57:57.161911 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20240502 20:57:57.251600 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240502 20:57:57.417387 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20240502 20:57:57.808903 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20240502 20:57:57.863474 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:57:57.900552 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:57:57.913952 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:57:58.089934 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:58.164022 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:57:58.227280 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:57:58.296694 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 20:57:58.588101 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:57:58.697412 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:58.718711 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:57:58.765887 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
I20240502 20:57:58.954205 19164 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:57:58.954403 19164 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:57:58.954932 19164 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:57:58.955580 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:57:58.955878 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:57:58.956408 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:57:58.956781 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:58.956903 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:57:58.957175 19164 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:57:59.007618 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20240502 20:57:59.076014 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:57:59.254894 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20240502 20:57:59.263607 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 20:57:59.319139 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:57:59.372982 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20240502 20:57:59.430830 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:57:59.778641 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20240502 20:57:59.959120 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
I20240502 20:57:59.960029 19164 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:57:59.960194 19164 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:57:59.960645 19164 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:57:59.962121 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:57:59.962379 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:57:59.962478 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:57:59.962742 19164 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:00.108557 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:58:00.202821 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:58:00.365335 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:58:00.377009 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:58:00.462317 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240502 20:58:00.476902 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:58:00.535888 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20240502 20:58:00.575630 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:00.771111 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:58:01.205437 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:58:01.216034 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:58:01.259630 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20240502 20:58:01.462858 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:01.463191 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:58:01.493299 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
I20240502 20:58:01.584847 19221 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:01.585019 19221 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:58:01.585637 19221 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:58:01.586099 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:58:01.586266 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:58:01.586746 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:58:01.586856 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:01.586907 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:58:01.587144 19221 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:01.719852 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:58:01.758721 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 20:58:01.775796 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:01.946995 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:58:02.021199 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:58:02.313694 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240502 20:58:02.349050 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
I20240502 20:58:02.480010 19176 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:02.480232 19176 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:58:02.480863 19176 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:58:02.481444 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:58:02.481626 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:58:02.482110 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:58:02.482635 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:02.482715 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:58:02.482870 19176 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 20:58:02.552327 19290 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:58:02.552496 19290 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:58:02.553136 19290 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:58:02.553562 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:58:02.554014 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:58:02.556294 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:58:02.556466 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:58:02.556552 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:02.557284 19290 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:58:02.643571 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:02.655560 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:58:02.784627 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:02.837905 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:58:02.957331 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:58:03.002668 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:03.039166 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:58:03.342242 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:58:03.462971 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 20:58:03.674046 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:58:03.788177 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:58:03.937366 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:04.122390 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:58:04.147733 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:04.161813 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:58:04.340433 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:04.342413 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:04.406416 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 20:58:04.409137 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:04.441524 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:58:04.669939 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20240502 20:58:04.918262 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:05.043902 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:05.238646 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:58:05.291457 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:58:05.370040 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:58:05.375655 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:05.401213 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:05.495750 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:58:05.530555 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:58:05.651193 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 20:58:06.130221 19574 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:06.130436 19574 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:58:06.131105 19574 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:58:06.131691 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:58:06.131901 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:58:06.132586 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:58:06.132699 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:58:06.132781 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:58:06.132930 19574 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:06.141744 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:06.340174 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:58:06.479755 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:58:06.682987 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20240502 20:58:06.712381 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
I20240502 20:58:06.724644 19587 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:58:06.724968 19587 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:58:06.726045 19587 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:58:06.726778 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:58:06.727777 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:58:06.729152 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:06.729310 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:58:06.730090 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:58:06.730212 19587 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:58:06.743263 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:06.934139 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:06.937866 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:07.003557 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 20:58:07.004541 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:58:07.070593 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:07.070603 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:07.329037 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240502 20:58:07.693424 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:58:07.710289 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:07.718935 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:07.725572 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:07.738857 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:58:07.995285 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:58:08.029690 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:08.150983 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:58:08.240564 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:08.477211 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 20:58:08.828081 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:58:08.863598 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:58:09.117136 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:09.121768 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20240502 20:58:09.244370 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:09.315125 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:09.352738 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:09.362634 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:09.750535 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:09.756596 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:58:09.809027 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 20:58:09.969786 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:58:09.984776 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:10.084349 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:58:10.185322 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:58:10.356268 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:10.390401 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:58:10.515487 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:10.547140 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:58:10.548177 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:10.579361 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:58:10.853828 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:10.980928 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:58:11.526156 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:58:11.579663 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:11.655354 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20240502 20:58:11.664731 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:58:11.783332 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:11.891134 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:11.895315 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:11.977303 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:12.043547 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
I20240502 20:58:12.309590 19872 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:12.309805 19872 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:58:12.310467 19872 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:58:12.311249 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:58:12.311440 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:58:12.311889 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:58:12.312081 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:12.312139 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:58:12.312410 19872 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:12.357358 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:58:12.370262 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240502 20:58:12.454800 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:58:12.511684 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:12.548691 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:58:12.641175 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 20:58:12.854945 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:12.873910 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:58:13.132464 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:13.281123 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:13.390671 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:13.636983 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:58:13.704481 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 20:58:13.846292 19872 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:13.846515 19872 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:58:13.847213 19872 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:58:13.847929 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:58:13.848188 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:58:13.848806 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:58:13.848945 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:13.848994 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:58:13.849265 19872 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:13.969149 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:58:14.076696 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:14.127521 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:58:14.297544 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20240502 20:58:14.327360 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:14.332528 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:14.382789 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
I20240502 20:58:14.520550 19872 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:58:14.520802 19872 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:58:14.521397 19872 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:58:14.523087 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:58:14.523236 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:14.523370 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:58:14.523605 19872 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:14.683794 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:14.765916 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:15.014582 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:15.046483 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:58:15.152443 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:58:15.180258 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:58:15.291899 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 20:58:15.515959 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:58:15.559656 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:15.616973 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 20:58:15.635519 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:15.636178 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:58:15.637037 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:16.092213 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:16.317334 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:58:16.413125 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:58:16.512997 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:16.520637 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:16.617146 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:58:16.701362 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:16.768641 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20240502 20:58:16.890775 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
I20240502 20:58:16.933884 20130 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:58:16.934170 20130 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:58:16.934979 20130 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:58:16.935613 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:58:16.935868 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:58:16.936580 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:16.936666 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:58:16.936760 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:58:16.936856 20130 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:58:17.158759 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:17.379155 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240502 20:58:17.512841 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:17.530411 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:17.537783 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:58:17.652176 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:58:17.662705 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 20:58:17.747167 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:58:17.872048 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:58:18.106742 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:18.113513 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:18.204865 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:18.238190 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
I20240502 20:58:18.545642 20096 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:18.545836 20096 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:58:18.546555 20096 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:58:18.547055 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:58:18.547331 27358 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:58:18.548522 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:58:18.548628 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:18.548740 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:58:18.548960 20096 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:18.724943 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:18.792136 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:58:18.929512 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:58:19.057495 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:58:19.171825 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 20:58:19.202282 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:19.308717 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20240502 20:58:19.359112 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:19.491677 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
I20240502 20:58:19.667129 20274 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:58:19.667398 20274 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:58:19.668057 20274 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:58:19.669507 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:58:19.669718 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:19.669816 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:58:19.670168 20274 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:19.739671 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:20.055620 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:58:20.104936 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 20:58:20.115211 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:20.120425 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:58:20.223306 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:20.269157 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:58:20.376159 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:58:20.523597 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:20.636402 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 20:58:20.675372 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 20:58:20.694725 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:20.817203 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:20.859103 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:21.072567 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:21.300209 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:58:21.329504 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:58:21.537869 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:58:21.732211 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:21.802695 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:21.804617 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20240502 20:58:21.837193 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:21.853924 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:22.403105 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240502 20:58:22.404259 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 20:58:22.483392 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:58:22.575215 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:58:22.629530 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:22.636356 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:58:22.702464 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:22.932534 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:58:22.986135 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:23.220917 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:23.356407 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:23.479727 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:23.627089 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:58:23.958765 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:23.974435 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:58:24.109436 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:24.189965 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:58:24.228336 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:24.270367 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:24.376717 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20240502 20:58:24.487309 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:24.524263 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:58:24.616132 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 20:58:24.799774 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:58:24.835000 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:58:24.958056 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:58:25.258663 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:25.308933 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:25.333256 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:25.507546 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:58:25.531121 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:25.670253 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:58:25.771224 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:58:25.913416 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:26.021047 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:58:26.191581 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:26.470561 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:58:26.525403 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:58:26.555630 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:26.722606 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:26.735122 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:26.774689 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:26.942526 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20240502 20:58:26.943744 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:26.946887 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 20:58:27.261276 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:58:27.302207 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:58:27.419903 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240502 20:58:27.473824 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:58:27.574527 20637 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:58:27.574721 20637 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:58:27.575455 20637 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:58:27.577184 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:58:27.577431 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:27.577548 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:58:27.577739 20637 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:27.827706 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:27.852316 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:27.978106 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:28.016842 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:58:28.019974 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:28.438163 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:28.508733 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:58:28.635993 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20240502 20:58:28.834785 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 20:58:28.915946 20718 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:28.916144 20718 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:58:28.916868 20718 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:58:28.917690 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:58:28.917968 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:58:28.918637 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:58:28.918800 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:28.918865 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:58:28.919082 20718 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:28.920800 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:28.981731 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20240502 20:58:29.010061 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:58:29.149955 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20240502 20:58:29.154306 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:29.236174 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20240502 20:58:29.265127 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 20:58:29.504788 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:58:29.510551 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:29.744434 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:58:29.851290 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:58:29.853015 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:58:30.299176 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:30.366351 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
I20240502 20:58:30.379504 20832 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:30.379802 20832 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:58:30.380681 20832 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:58:30.381386 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:58:30.381601 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:58:30.382609 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:58:30.382745 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:58:30.382810 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:58:30.383065 20832 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:30.458300 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:30.591862 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20240502 20:58:30.703073 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:58:30.725255 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:58:30.942245 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20240502 20:58:31.157147 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:31.217288 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:31.223558 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:31.278424 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:58:31.355217 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:58:31.439435 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:31.588233 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:58:31.706616 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:31.734593 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 20:58:31.775947 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:58:31.812057 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20240502 20:58:32.022686 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20240502 20:58:32.076886 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:58:32.238574 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
I20240502 20:58:32.306564 26799 maintenance_manager.cc:419] P e0c668105a124f0eb7227390002cc15e: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033394
I20240502 20:58:32.311273 26798 maintenance_manager.cc:643] P e0c668105a124f0eb7227390002cc15e: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.004s	user 0.002s	sys 0.000s Metrics: {"bytes_written":2453,"delete_count":0,"lbm_write_time_us":241,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":90,"threads_started":1,"update_count":39,"wal-append.queue_time_us":276}
I20240502 20:58:32.311906 26799 maintenance_manager.cc:419] P e0c668105a124f0eb7227390002cc15e: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.154588
I20240502 20:58:32.318447 26798 maintenance_manager.cc:643] P e0c668105a124f0eb7227390002cc15e: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.006s	user 0.006s	sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":16456,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":296,"lbm_read_time_us":173,"lbm_reads_lt_1ms":17,"lbm_write_time_us":247,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":147,"threads_started":2,"update_count":39}
W20240502 20:58:32.341617 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:58:32.421485 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240502 20:58:32.615945 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:32.732774 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20240502 20:58:32.910392 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:33.134624 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20240502 20:58:33.166347 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 20:58:33.346275 20890 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:58:33.346509 20890 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:58:33.347316 20890 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:58:33.347987 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:58:33.348475 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:58:33.348770 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:58:33.348932 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:33.349011 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:58:33.349220 20890 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:58:33.587395 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20240502 20:58:33.684504 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20240502 20:58:33.732630 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:33.756125 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:33.879812 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:58:33.915936 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:58:34.102766 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:58:34.111016 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:58:34.237188 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:34.296144 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 20:58:34.545531 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:34.598402 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20240502 20:58:34.643254 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:58:34.682166 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:34.977299 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:58:34.978570 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:58:35.116674 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:35.249742 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:35.434617 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:58:35.445283 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:35.675347 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:35.798815 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:58:35.799798 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:35.952207 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:58:36.072149 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:36.128688 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:36.153533 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:58:36.278489 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:36.440264 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:58:36.690452 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:58:36.779438 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 20:58:36.807766 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:58:36.825513 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:37.098800 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:37.241899 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20240502 20:58:37.301873 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:37.397809 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:58:37.442960 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240502 20:58:37.444036 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:58:37.479871 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:37.955590 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:58:37.958583 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:58:37.959767 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:58:38.093290 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:38.241746 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:38.253646 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:38.460170 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:38.579504 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:38.652237 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:38.715711 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:38.779537 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:58:38.821020 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:58:39.068688 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:58:39.117830 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:58:39.158221 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:58:39.245261 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 20:58:39.289361 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:39.467967 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:39.741469 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20240502 20:58:39.810879 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:39.819537 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 20:58:39.826565 20890 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:39.826746 20890 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:58:39.827332 20890 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:58:39.828087 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:58:39.828356 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:58:39.828943 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:58:39.829142 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:58:39.829209 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:58:39.829460 20890 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:40.033929 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:58:40.114063 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:40.128302 26796 catalog_manager.cc:4573] TS 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a2b61909194d488682afd91c0f796405: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:58:40.327448 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:58:40.343375 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:58:40.405912 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:40.648487 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
I20240502 20:58:40.654512 20890 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:40.654711 20890 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:58:40.655630 20890 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:58:40.656203 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:58:40.656422 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:58:40.657058 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:58:40.657240 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:40.657299 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:58:40.657445 20890 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:40.738827 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
I20240502 20:58:40.873662 20890 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:58:40.873844 20890 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:58:40.874454 20890 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:58:40.875936 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
W20240502 20:58:40.876274 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:40.876464 27295 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:58:40.876722 20890 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:40.902391 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:40.965384 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:58:41.071238 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:58:41.135571 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5586b9e60b544aed87c24f7b3ce0198c: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:58:41.135960 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:41.155727 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:41.203603 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:41.322746 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:58:41.396960 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:58:41.428095 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:58:41.516629 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:58:41.652454 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 20:58:41.680830 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:41.868577 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:42.128070 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20240502 20:58:42.179219 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:58:42.283129 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:58:42.453121 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 20:58:42.598378 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:42.680177 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:42.815933 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:58:42.872223 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a547561343304137a044ce11517f6007: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:58:43.044466 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:43.056676 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:43.345279 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:43.515676 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:43.569679 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:43.772974 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:58:43.784180 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:43.789994 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:43.890304 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:58:43.985698 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:58:44.104701 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:44.107780 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 20:58:44.240384 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:58:44.275213 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:44.461146 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20240502 20:58:44.654287 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 20:58:44.783344 21478 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:58:44.783547 21478 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:58:44.784184 21478 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:58:44.784539 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
W20240502 20:58:44.785506 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:58:44.785537 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:58:44.785959 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:58:44.786190 21478 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:58:44.786495 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:58:44.888386 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:58:45.050076 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:45.267787 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:45.306037 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:58:45.459766 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:45.685113 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:45.703068 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:58:45.743505 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:45.818151 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:45.855661 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:46.002178 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:58:46.154269 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:46.214565 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:58:46.218484 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:46.306280 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:58:46.334869 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:58:46.375263 21399 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:46.375468 21399 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:58:46.376221 21399 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:58:46.376842 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:58:46.377117 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:58:46.377444 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:58:46.377664 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:46.377765 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:58:46.377908 21399 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:46.498633 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 20:58:46.727386 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:58:46.797477 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:46.810204 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:46.852826 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 20:58:47.013186 21685 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:47.013424 21685 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:58:47.014075 21685 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:58:47.014743 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:58:47.014999 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:58:47.015559 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:58:47.015661 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:47.015712 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:58:47.015877 21685 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:47.070755 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20240502 20:58:47.293644 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:58:47.460045 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 20:58:47.460974 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:47.720738 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:58:47.814023 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:47.933154 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:58:48.128047 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:48.205773 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:48.211331 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:48.271389 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:48.414379 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:48.592154 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:48.773695 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:48.819576 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:58:48.939486 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:58:49.155225 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:58:49.160143 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 20:58:49.268822 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:49.305429 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:58:49.315227 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
I20240502 20:58:49.500260 21839 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:58:49.500501 21839 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:58:49.501448 21839 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:58:49.501946 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:58:49.502287 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:58:49.502954 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:58:49.503162 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:58:49.503264 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 1 yes votes; 4 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:58:49.503437 21839 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:58:49.641248 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20240502 20:58:49.646338 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:58:50.084775 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:50.173899 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:58:50.416517 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:50.429634 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:58:50.759881 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:50.768810 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:50.906984 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:51.009994 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:51.016055 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 20:58:51.040474 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:58:51.154779 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
I20240502 20:58:51.200701 21918 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:51.200934 21918 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:58:51.201601 21918 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:58:51.202585 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:58:51.202785 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:58:51.203078 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:58:51.203244 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:51.203294 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:58:51.203478 21918 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:51.209754 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:58:51.227614 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:51.286862 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:51.295212 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 46 similar messages]
I20240502 20:58:51.589042 21955 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:58:51.589368 21955 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:58:51.590009 21955 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:58:51.591477 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:58:51.591743 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:58:51.591822 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:58:51.592070 21955 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:51.625854 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:58:51.708729 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:58:51.721757 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:51.773792 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 20:58:51.795013 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:51.808501 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:58:51.826144 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:58:52.336187 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20240502 20:58:52.534044 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240502 20:58:52.535014 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:58:52.767159 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:53.017419 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:58:53.152215 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:53.195403 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:53.246889 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:53.263818 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:53.554350 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:58:53.628688 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:53.651006 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:53.726796 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:53.891353 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:58:53.936182 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:54.022356 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:54.041568 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:54.180824 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:58:54.263413 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:58:54.432035 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 20:58:54.711498 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:58:55.031124 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20240502 20:58:55.044935 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:58:55.355989 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:55.378113 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:55.640163 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:58:55.751498 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:55.953399 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:55.962174 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:58:56.041144 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 20:58:56.180130 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:58:56.203359 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:56.281476 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:58:56.341231 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:56.351116 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:58:56.352298 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:58:56.474826 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:58:56.535607 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:58:56.551625 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:56.578964 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:56.598330 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:58:56.926743 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:58:56.958119 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 20:58:57.047540 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:58:57.478922 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20240502 20:58:57.505466 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:58:57.574609 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 108 similar messages]
I20240502 20:58:57.787770 22067 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:57.787983 22067 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:58:57.788697 22067 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:58:57.789201 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:58:57.789358 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:58:57.790205 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:58:57.790745 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:58:57.790827 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:58:57.790983 22067 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:58:57.841020 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:57.853739 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:58:58.254518 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:58.282119 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:58:58.509769 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:58:58.535020 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:58:58.573109 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:58:58.601459 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:58.659883 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:58:58.766415 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:58.921420 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:58:58.936686 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:58:59.216473 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:58:59.235150 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:58:59.320269 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:58:59.332475 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:58:59.478404 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 20:58:59.551841 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:58:59.834204 22252 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:58:59.834450 22252 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:58:59.835094 22252 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:58:59.835587 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:58:59.835752 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:58:59.836979 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:58:59.837178 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:58:59.837250 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:58:59.837438 22252 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:00.058925 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:59:00.096769 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20240502 20:59:00.290340 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:59:00.581930 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:59:00.760885 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 20:59:00.766989 22252 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:59:00.767172 22252 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:59:00.767623 22252 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:59:00.768939 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:00.769091 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:59:00.769160 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:59:00.769325 22252 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:00.909586 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:59:00.924714 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:59:01.013317 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:59:01.089712 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:59:01.091003 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:01.202535 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
I20240502 20:59:01.206521 22252 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:59:01.206753 22252 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:59:01.207746 22252 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:59:01.208138 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:59:01.209261 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:59:01.209692 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:59:01.209793 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:01.209897 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:59:01.210274 22252 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:59:01.311978 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:59:01.322399 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:59:01.350711 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:59:01.633518 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 20:59:01.644205 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:59:01.645776 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:59:01.716421 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:59:01.744995 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:01.761576 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:59:01.974045 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:59:01.985692 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 20:59:02.060822 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:59:02.623420 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240502 20:59:02.856324 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:02.884994 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:59:02.919386 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20240502 20:59:03.034513 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
I20240502 20:59:03.035179 22537 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:03.035326 22537 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:59:03.036082 22537 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:59:03.036594 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:59:03.036834 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:59:03.037453 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:03.037576 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:03.037629 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:59:03.037910 22537 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:03.075893 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:59:03.125139 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 20:59:03.243291 22537 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:03.243553 22537 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:59:03.244262 22537 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:59:03.245141 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:59:03.245405 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:59:03.245900 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:59:03.246258 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:03.246359 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:59:03.246639 22537 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:03.367003 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:03.499684 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:59:03.608306 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:59:03.737511 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:03.783083 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:03.807520 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:59:03.935411 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:59:04.120972 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:59:04.254256 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:04.413501 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:59:04.430964 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 20:59:04.566854 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:04.599993 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:59:04.643973 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:59:05.360749 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:05.443430 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:59:05.495465 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20240502 20:59:05.567929 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:05.610792 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:59:05.649545 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:59:05.890115 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:06.101313 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:06.127678 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:06.146526 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:06.184571 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:59:06.185842 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:06.231443 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:59:06.310740 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:06.387033 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:59:06.716488 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 20:59:06.820101 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:59:06.839867 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:06.908973 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:59:07.052650 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:07.143508 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:59:07.145666 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:07.314193 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 20:59:07.669549 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 20:59:07.801313 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:59:07.825438 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:07.999353 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20240502 20:59:08.111174 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:59:08.139247 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:08.281304 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:08.317565 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:59:08.454605 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:08.536980 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:08.574021 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:08.601186 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:59:08.609205 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:08.797688 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:59:08.880409 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:09.197777 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:09.317776 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:59:09.345679 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:09.692332 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:09.969673 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:59:10.015971 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 20:59:10.093427 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 20:59:10.282227 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 20:59:10.429042 22900 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:10.429315 22900 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:59:10.429942 22900 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:59:10.430750 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:59:10.431093 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:59:10.431527 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:59:10.431699 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:10.431798 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:59:10.432139 22900 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:10.488874 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20240502 20:59:10.528195 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:10.633637 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:59:10.687176 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:10.745795 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:59:10.803313 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:59:10.820040 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:10.865295 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:10.949558 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:10.980818 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:11.197600 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:11.258002 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:59:11.330763 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:11.426692 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:59:11.660579 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:11.675271 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:11.800966 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:59:11.842473 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:59:12.055953 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:59:12.102182 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:12.300891 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:59:12.524070 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 20:59:12.710023 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240502 20:59:12.866287 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:59:13.039566 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:13.095544 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:59:13.146610 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20240502 20:59:13.160526 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:59:13.189651 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:13.202613 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:13.315558 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:13.375869 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:13.382339 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:59:13.587507 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:13.644449 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:13.716655 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:13.862329 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:59:14.166951 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:14.232841 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:14.269150 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:59:14.555703 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:14.810897 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:59:14.972229 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 20:59:15.226505 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:59:15.364554 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:15.546514 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:15.559181 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20240502 20:59:15.564070 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:15.679301 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:59:15.741155 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:59:15.808990 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:59:15.816182 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:15.889597 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
I20240502 20:59:16.093966 23068 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:59:16.094223 23068 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:59:16.095044 23068 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:59:16.095463 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:59:16.096130 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:59:16.096731 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:16.096944 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:59:16.097182 23068 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:59:16.097057 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:59:16.110327 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:16.258821 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:16.267501 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:16.333211 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 20:59:16.387346 22447 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:16.387590 22447 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:59:16.388509 22447 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:59:16.389194 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:59:16.389433 27358 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:59:16.390867 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:59:16.390990 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:16.391047 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:59:16.391240 22447 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:16.574313 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:59:16.671270 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:16.678437 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:59:16.755836 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
I20240502 20:59:16.814699 23168 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:16.814909 23168 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:59:16.815584 23168 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:59:16.816413 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:59:16.816622 27358 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:59:16.817337 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:16.817508 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:16.817569 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:59:16.817785 23168 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:16.857740 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:59:16.988818 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:17.120440 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 20:59:17.537570 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:59:17.672547 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 20:59:17.687678 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:59:17.783020 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240502 20:59:17.870116 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:17.918292 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:18.055867 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:59:18.180686 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:18.197688 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:59:18.221977 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:18.227948 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:59:18.268498 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20240502 20:59:18.371502 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:18.453660 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
I20240502 20:59:18.845373 23268 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:59:18.845579 23268 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:59:18.846151 23268 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:59:18.847687 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:18.847904 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:59:18.848008 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:59:18.848201 23268 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:18.889052 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:18.939489 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:19.023845 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:19.042579 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:59:19.221457 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:19.257786 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 20:59:19.587167 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:19.979319 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:59:20.033759 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:59:20.170706 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 20:59:20.210561 23418 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:20.210798 23418 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:59:20.212834 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:59:20.213165 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
I20240502 20:59:20.213181 23418 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
W20240502 20:59:20.213891 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:20.214835 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:20.214923 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:59:20.215204 23418 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:20.331192 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:20.343021 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:20.482180 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:59:20.504912 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:20.646488 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:20.699430 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:59:20.797176 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:59:20.920881 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:20.959971 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:21.036165 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20240502 20:59:21.321363 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:21.327633 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:59:21.368746 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:59:21.443043 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:21.476821 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:21.731042 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:59:21.889380 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:21.892505 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240502 20:59:22.079077 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:22.179047 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:59:22.384088 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 20:59:22.445576 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:59:22.691975 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:59:22.732800 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:22.759568 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:22.868654 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240502 20:59:22.869589 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:59:22.936997 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:23.134999 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:23.144380 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:59:23.259346 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:59:23.281689 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20240502 20:59:23.529630 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:23.554296 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:23.629276 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:23.675653 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:23.807248 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:59:23.963304 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:24.205078 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:59:24.332886 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:24.741670 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:24.857049 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 20:59:25.023160 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:59:25.186604 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:25.334901 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 20:59:25.337002 23620 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:25.337368 23620 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:59:25.337926 23620 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:59:25.338434 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:59:25.338621 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:59:25.339195 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:25.339679 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:59:25.339766 27296 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:59:25.339931 23620 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:25.346848 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:25.545459 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:59:25.571610 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:25.629751 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20240502 20:59:25.635986 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:25.694005 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:59:25.798030 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:59:25.905503 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:25.984691 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:26.130601 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:26.381126 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:59:26.403901 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:26.579703 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:59:26.602417 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
I20240502 20:59:26.609371 23620 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:59:26.609619 23620 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:59:26.610435 23620 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:59:26.611874 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:26.612149 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:59:26.612277 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:59:26.612483 23620 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:26.746922 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:59:26.865051 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:27.067737 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240502 20:59:27.343243 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 20:59:27.468825 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:59:27.526777 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:27.579173 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:59:27.587904 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:59:27.618531 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:27.662287 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:59:27.892199 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 110 similar messages]
I20240502 20:59:27.952227 23620 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:27.952454 23620 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:59:27.953227 23620 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:59:27.953825 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:59:27.954082 27091 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:59:27.954761 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:27.954895 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:27.954942 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:59:27.955192 23620 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:28.004859 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:28.061733 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:28.081669 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20240502 20:59:28.128140 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:59:28.135162 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:28.157235 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:59:28.157894 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:28.401564 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:28.406790 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:28.627939 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:28.920955 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:59:29.211248 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:29.214541 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:59:29.698508 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:29.884776 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 20:59:29.943456 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:30.112300 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:59:30.112716 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:59:30.341403 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:59:30.364488 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:30.483960 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:30.618281 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:30.708971 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:30.712575 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:59:30.744915 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:59:30.762954 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:30.763413 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:30.788398 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20240502 20:59:30.821624 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:31.049301 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:31.537412 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:59:31.665370 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:59:31.709414 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 20:59:31.851708 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:32.122574 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:59:32.158869 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:32.185050 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 20:59:32.417201 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:32.521406 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:59:32.591562 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:59:32.613675 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:59:32.825979 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:32.890805 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:32.905788 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 114 similar messages]
W20240502 20:59:32.906831 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:59:33.005375 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:59:33.061579 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:33.104123 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:33.114352 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20240502 20:59:33.146544 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:33.176255 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:33.221750 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
I20240502 20:59:33.272729 24090 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:59:33.272980 24090 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:59:33.273919 24090 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:59:33.274631 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:59:33.275403 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:59:33.275599 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:33.275787 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:59:33.275876 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 20:59:33.276062 24090 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:59:33.395607 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:59:33.517009 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
I20240502 20:59:33.579231 24090 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:59:33.579519 24090 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:59:33.580180 24090 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:59:33.581718 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:33.582516 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:59:33.582646 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:59:33.582931 24090 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:34.154862 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:59:34.218724 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:59:34.523777 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:34.536260 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:34.643301 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 20:59:34.666471 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:34.923663 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:59:35.215232 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:59:35.305907 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:59:35.422591 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:35.477909 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:35.480441 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:59:35.555485 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:35.566573 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
I20240502 20:59:35.624554 24250 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:35.624742 24250 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:59:35.625468 24250 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:59:35.626225 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:59:35.626500 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:59:35.626654 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:35.626998 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:59:35.627068 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:59:35.627303 24250 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:35.651305 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:59:35.676384 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:35.691190 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:35.721573 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20240502 20:59:35.749603 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:35.983759 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:36.684561 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:59:36.743077 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:59:36.909281 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:36.933591 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:59:37.005416 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:37.244372 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:37.289093 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240502 20:59:37.363723 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 20:59:37.375361 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:59:37.652081 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:59:37.674580 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 20:59:37.722373 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:59:37.788635 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:59:37.905537 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:37.928581 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 20:59:37.929872 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:37.944152 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:59:37.977881 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 20:59:37.978940 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f00d12e425284c36ae8baed56c7cfaf6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:37.981447 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:38.006652 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:38.049647 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:38.106725 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20240502 20:59:38.196780 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:38.274227 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:38.433344 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
I20240502 20:59:38.786863 24274 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:38.787070 24274 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:59:38.787770 24274 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:59:38.788282 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:59:38.788470 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:59:38.789151 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:38.789276 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:38.789345 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:59:38.789505 24274 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:39.151839 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:59:39.167462 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a8ce1df86914b2caf2d083d97eda4e6: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:39.310611 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:39.320794 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:39.338873 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:59:39.683642 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 20:59:39.821228 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:40.047783 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:59:40.195631 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:59:40.204843 26794 catalog_manager.cc:4573] TS 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f1545142b0bd40859bf33bf8686002ae: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:40.273020 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:59:40.362569 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20240502 20:59:40.395112 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:40.453038 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:59:40.572240 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20240502 20:59:40.587623 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:40.614054 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:40.616748 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:59:40.730784 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:40.820894 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:40.883510 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:40.899910 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20240502 20:59:41.251418 26795 catalog_manager.cc:4573] TS 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 62ebc6576a4e4a108de55a4e9d7ff6b8: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:59:41.549408 24481 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:59:41.549587 24481 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:59:41.550134 24481 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:59:41.551347 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:41.551457 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:59:41.551576 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:59:41.551740 24481 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:41.639336 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:59:41.771456 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20240502 20:59:41.854456 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:59:41.990268 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:42.014973 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:59:42.025693 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 20:59:42.409737 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 20:59:42.497104 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:59:42.500353 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:59:42.556021 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:42.783838 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 20:59:42.912672 26794 catalog_manager.cc:4573] TS 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a547561343304137a044ce11517f6007: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:42.916837 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:42.937146 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:59:42.943383 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:59:42.957962 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240502 20:59:43.085038 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20240502 20:59:43.094823 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:59:43.136224 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:43.170800 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:43.227252 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:43.486526 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:43.507818 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:43.575919 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20240502 20:59:43.582791 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20240502 20:59:44.207259 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:59:44.359169 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:59:44.391422 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 20:59:44.414964 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:44.767145 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20240502 20:59:44.928820 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:44.952122 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:59:45.116389 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:59:45.257195 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
I20240502 20:59:45.272238 24712 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:45.272461 24712 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:59:45.273116 24712 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:59:45.273793 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:59:45.274029 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:59:45.274896 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:45.275123 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:45.275231 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:59:45.275439 24712 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:45.326192 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:59:45.394551 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:59:45.407157 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20240502 20:59:45.579005 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:59:45.724535 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20240502 20:59:45.727510 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20240502 20:59:45.759465 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20240502 20:59:45.814220 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
I20240502 20:59:45.997222 24677 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:45.997421 24677 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:59:45.998019 24677 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:59:45.998517 26957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
I20240502 20:59:45.998714 26957 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 20:59:45.999325 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 20:59:45.999464 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:45.999543 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:59:45.999701 24677 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:46.059370 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20240502 20:59:46.071259 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:46.439105 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:46.743245 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:59:46.836117 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:59:46.924513 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20240502 20:59:47.050084 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:59:47.184276 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 20:59:47.245172 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:47.350291 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:59:47.508755 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:59:47.536283 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:47.548342 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:59:47.557037 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
I20240502 20:59:47.665347 24712 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:47.665526 24712 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:59:47.666033 24712 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:59:47.666599 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:59:47.666779 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:59:47.667411 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:47.667548 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:47.667603 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:59:47.667797 24712 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:47.906435 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:59:47.972226 27025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240502 20:59:47.997808 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:59:48.065663 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:59:48.067909 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:48.096630 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:48.165539 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:59:48.178121 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:48.221057 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:48.267601 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:48.300333 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20240502 20:59:48.562867 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20240502 20:59:49.013816 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:59:49.124229 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20240502 20:59:49.145989 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:59:49.476629 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 20:59:49.661377 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:49.690313 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20240502 20:59:49.987988 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:50.045820 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:50.070605 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:59:50.138532 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:59:50.357568 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:59:50.667906 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20240502 20:59:50.673779 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20240502 20:59:50.695643 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:59:50.709233 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:59:50.733850 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20240502 20:59:50.814836 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:50.896600 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
I20240502 20:59:50.922747 24963 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:50.922945 24963 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 20:59:50.923672 24963 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 20:59:50.924669 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:59:50.924865 27358 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 20:59:50.924937 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:50.925400 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:59:50.925473 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 20:59:50.925626 24963 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:50.978164 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20240502 20:59:50.980212 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:51.549134 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:51.654402 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:59:51.753562 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 20:59:51.795370 24975 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 20:59:51.795593 24975 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 20:59:51.796416 24975 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 20:59:51.796981 27091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 20:59:51.797158 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 20:59:51.798067 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 20:59:51.798164 27296 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 297de823852d4a8da44791ab03131cd3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
W20240502 20:59:51.798269 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:51.798362 24975 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 20:59:52.111668 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 20:59:52.198803 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:52.212389 27160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 20:59:52.213562 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20240502 20:59:52.316715 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:52.410444 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:59:52.426254 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 20:59:52.661650 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 20:59:52.705736 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:59:52.981141 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240502 20:59:53.010715 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:59:53.036937 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:53.189509 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 20:59:53.200781 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:53.225610 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20240502 20:59:53.435241 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:53.496495 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:53.520432 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:59:53.536510 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:59:53.592967 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:53.610107 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 20:59:54.129439 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:54.164163 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:59:54.164163 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:59:54.646747 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 20:59:54.647773 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 20:59:54.903909 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:54.916167 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:54.955333 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:59:55.082571 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:59:55.177186 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 20:59:55.579449 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:59:55.635522 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:55.636739 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:55.859411 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:59:55.885161 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:55.922735 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:55.974361 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20240502 20:59:56.002115 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 20:59:56.035197 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:59:56.068104 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
I20240502 20:59:56.214774 25123 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 20:59:56.214955 25123 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 20:59:56.215497 25123 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 20:59:56.217322 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 20:59:56.219450 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 20:59:56.219539 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 20:59:56.219897 25123 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:56.596150 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:59:56.672459 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 20:59:56.686601 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:59:57.066085 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 20:59:57.078387 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 20:59:57.263659 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 20:59:57.417443 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
W20240502 20:59:57.477618 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 20:59:57.523666 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:59:57.577478 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 20:59:57.605571 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:59:57.807801 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 20:59:57.987375 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 110 similar messages]
[New LWP 19725]
[New LWP 19726]
[New LWP 26769]
[New LWP 26770]
[New LWP 26771]
[New LWP 26772]
[New LWP 26773]
[New LWP 27951]
[New LWP 27952]
[New LWP 27953]
[New LWP 27954]
[New LWP 29302]
W20240502 20:59:58.175531 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20240502 20:59:58.224779 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 20:59:58.237807 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 20:59:58.266144 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 20:59:58.334693 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:59:58.447575 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:59:58.487752 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 20:59:58.514324 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 20:59:58.535959 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 20:59:58.606786 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 20:59:58.846398 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 20:59:58.957398 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 20:59:59.013001 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 20:59:59.141165 25354 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 20:59:59.141346 25354 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 20:59:59.142022 25354 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 20:59:59.142544 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 20:59:59.142724 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 20:59:59.143918 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 20:59:59.144127 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 20:59:59.144202 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 20:59:59.144376 25354 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 20:59:59.207160 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:59:59.484447 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 20:59:59.558530 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 20:59:59.842521 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 20:59:59.884538 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 20:59:59.916672 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:00:00.059262 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:00.273059 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:00:00.619995 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 21:00:00.653606 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:00:00.672255 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20240502 21:00:00.719368 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 21:00:00.757025 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:00.810938 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:00:00.957919 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:00:01.323624 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 21:00:01.457904 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 21:00:01.510564 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:00:01.538985 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 21:00:01.613886 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 21:00:01.671803 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:00:01.854249 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:02.054955 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 21:00:02.067451 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 21:00:02.359107 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:00:02.393473 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:00:02.448712 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:02.450145 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20240502 21:00:02.669559 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:00:02.926008 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 21:00:02.998802 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:00:03.064885 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 21:00:03.076555 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 21:00:03.251844 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20240502 21:00:03.262620 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:03.292794 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 21:00:03.399004 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:00:03.451089 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 21:00:03.582466 25641 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:00:03.582648 25641 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 21:00:03.583317 25641 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 21:00:03.584794 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 21:00:03.585179 27296 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 21:00:03.585284 27295 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 21:00:03.585340 27295 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 21:00:03.585382 26960 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
I20240502 21:00:03.585527 25641 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:00:03.848706 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:00:03.850029 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:00:03.956432 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:03.994410 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 21:00:04.037498 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 21:00:04.079939 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:04.142813 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:04.168785 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 21:00:04.188786 25641 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 21:00:04.189018 25641 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 21:00:04.189818 25641 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 21:00:04.190609 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 21:00:04.191126 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 21:00:04.191576 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 21:00:04.192025 27298 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 297de823852d4a8da44791ab03131cd3, d1f3f8e444364d6faa42d651b5dbb997
W20240502 21:00:04.192157 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 21:00:04.192258 25641 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:00:04.465242 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:04.799259 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 21:00:04.885298 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:00:04.958268 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:04.980409 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 21:00:05.163295 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:00:05.321471 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:00:05.553578 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:05.627159 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:05.726523 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:00:05.885084 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:06.008644 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:00:06.021705 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20240502 21:00:06.360319 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:06.377940 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:00:06.382725 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:06.429905 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:06.561882 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:06.638880 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:00:06.763053 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:06.966373 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:07.080478 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:07.330849 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 21:00:07.503515 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:00:07.611590 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:00:07.646934 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:00:07.724658 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:07.842846 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:08.022130 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 21:00:08.022960 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:00:08.066583 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:00:08.090392 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240502 21:00:08.091378 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:08.224931 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
I20240502 21:00:08.320623 25850 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:00:08.320916 25850 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 21:00:08.321770 25850 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 21:00:08.323082 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 21:00:08.323308 27361 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 21:00:08.323688 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 21:00:08.323807 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
I20240502 21:00:08.323879 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 21:00:08.324021 25850 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:00:08.565712 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:00:08.733860 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:00:08.734256 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20240502 21:00:08.860982 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:08.867880 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:09.057303 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:09.177433 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:00:09.337229 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:09.338768 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:09.343935 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:09.364635 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:00:09.931206 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:10.004925 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:00:10.056887 26797 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet ff9fef57f6eb490da536170a02b96212 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }.
  Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
W20240502 21:00:10.066766 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:00:10.073776 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 21:00:10.272024 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:10.448283 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:10.507416 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:00:10.609993 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:10.700368 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:10.732416 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:00:11.267136 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:00:11.357897 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:11.359122 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20240502 21:00:11.385591 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:11.385685 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:00:11.506387 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:11.597988 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:11.769731 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:11.913256 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:11.994282 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:00:12.470331 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:12.542981 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:00:12.546104 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:00:12.616776 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 21:00:12.662039 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:00:12.766525 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:00:12.804270 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:00:12.992399 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:12.997568 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:13.072132 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:13.099359 27296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 21:00:13.104054 27024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240502 21:00:13.104753 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:13.445978 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:00:13.579128 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
I20240502 21:00:13.607977 25885 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:00:13.608156 25885 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 21:00:13.608814 25885 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 21:00:13.609700 27361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 21:00:13.609889 27361 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 21:00:13.610219 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 21:00:13.610303 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 21:00:13.610359 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 21:00:13.610630 26238 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:00:13.654518 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20240502 21:00:13.903335 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:13.916733 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:13.964026 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:00:14.024842 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:14.035660 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:14.401003 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 21:00:14.517601 26157 raft_consensus.cc:482] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1bc8a142cfc7487bacc573afb0a92b74)
I20240502 21:00:14.517763 26157 raft_consensus.cc:504] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "1bc8a142cfc7487bacc573afb0a92b74" member_type: VOTER last_known_addr { host: "127.19.66.135" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: NON_VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: true } }
I20240502 21:00:14.518244 26157 leader_election.cc:288] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845)
W20240502 21:00:14.519194 27295 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 21:00:14.519309 27297 leader_election.cc:334] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111)
I20240502 21:00:14.519380 27297 leader_election.cc:302] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1bc8a142cfc7487bacc573afb0a92b74
I20240502 21:00:14.519532 26157 raft_consensus.cc:2726] T 8deaa1f751034c15acd6ccaad37536c6 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:00:14.548224 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 21:00:14.623028 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:14.837744 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:00:14.918222 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:15.051879 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:00:15.121841 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:00:15.326452 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 21:00:15.466276 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:15.476915 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:15.507465 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:15.522349 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:15.588812 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:00:15.792877 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:00:15.991644 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:16.061956 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20240502 21:00:16.380409 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:16.399799 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:16.481632 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:16.668659 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:16.714624 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:00:16.798770 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 21:00:16.941404 26238 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:00:16.941597 26238 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 21:00:16.942201 26238 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 21:00:16.942857 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 21:00:16.943205 27358 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 21:00:16.943575 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 21:00:16.944703 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 21:00:16.944789 26891 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 21:00:16.945120 26238 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:00:17.163591 26307 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:00:17.163806 26307 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 21:00:17.164570 26307 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 21:00:17.165227 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "d1f3f8e444364d6faa42d651b5dbb997" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 21:00:17.165416 27358 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f3f8e444364d6faa42d651b5dbb997 in term 2.
W20240502 21:00:17.166033 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 21:00:17.166188 27024 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 21:00:17.166260 27024 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 21:00:17.166420 26307 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:00:17.425518 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:17.472638 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:00:17.549791 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:17.602800 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:00:17.745899 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:17.752912 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 21:00:17.827190 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:00:17.909240 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:00:17.937708 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:18.027979 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:18.105401 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 21:00:18.174489 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:18.219945 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 21:00:18.257746 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:00:18.319918 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:18.355827 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20240502 21:00:18.486234 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:00:18.839210 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:18.854099 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:18.964046 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:19.176368 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:00:19.199556 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
I20240502 21:00:19.309360 26238 raft_consensus.cc:482] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:00:19.309666 26238 raft_consensus.cc:504] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } }
I20240502 21:00:19.310532 26238 leader_election.cc:288] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd8300729ec4226acc38b9cff2d8d3b (127.19.66.132:41007), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529)
I20240502 21:00:19.311084 27358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a7bead40fb4eb0bec6b00804b52595" candidate_uuid: "297de823852d4a8da44791ab03131cd3" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" is_pre_election: true
I20240502 21:00:19.311291 27358 raft_consensus.cc:2454] T 14a7bead40fb4eb0bec6b00804b52595 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297de823852d4a8da44791ab03131cd3 in term 2.
W20240502 21:00:19.312367 26891 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
W20240502 21:00:19.312943 26892 leader_election.cc:334] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
I20240502 21:00:19.313041 26892 leader_election.cc:302] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 297de823852d4a8da44791ab03131cd3, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31
I20240502 21:00:19.313292 26238 raft_consensus.cc:2726] T 14a7bead40fb4eb0bec6b00804b52595 P 297de823852d4a8da44791ab03131cd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:00:19.316604 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:00:19.827618 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:00:19.901324 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:19.922281 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240502 21:00:19.926638 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:00:20.047159 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:20.343147 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:20.349606 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 21:00:20.465286 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:00:20.493466 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:20.533928 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:20.642540 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:20.651144 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20240502 21:00:20.732689 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:20.754012 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:00:20.791635 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:00:21.250914 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:21.393357 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:21.551261 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:21.600513 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:21.730206 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 21:00:21.734861 26556 raft_consensus.cc:482] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1f3f8e444364d6faa42d651b5dbb997)
I20240502 21:00:21.735221 26556 raft_consensus.cc:504] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } attrs { replace: true } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } } peers { permanent_uuid: "6706d1c0946d42a9bae62d719e1e5c31" member_type: VOTER last_known_addr { host: "127.19.66.136" port: 41529 } attrs { promote: false } } peers { permanent_uuid: "297de823852d4a8da44791ab03131cd3" member_type: VOTER last_known_addr { host: "127.19.66.129" port: 38835 } attrs { promote: false } }
I20240502 21:00:21.737044 26556 leader_election.cc:288] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737), 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529), 297de823852d4a8da44791ab03131cd3 (127.19.66.129:38835)
I20240502 21:00:21.737336 27089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 21:00:21.738163 26960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9fef57f6eb490da536170a02b96212" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "297de823852d4a8da44791ab03131cd3" is_pre_election: true
W20240502 21:00:21.738462 27296 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111)
W20240502 21:00:21.738605 27295 leader_election.cc:334] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 21:00:21.738662 27295 leader_election.cc:302] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 6706d1c0946d42a9bae62d719e1e5c31, d1f3f8e444364d6faa42d651b5dbb997
I20240502 21:00:21.738837 26556 raft_consensus.cc:2726] T ff9fef57f6eb490da536170a02b96212 P ebd8300729ec4226acc38b9cff2d8d3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:00:22.187667 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:00:22.198626 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:00:22.314121 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:00:22.452739 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:22.554337 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20240502 21:00:22.811232 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20240502 21:00:22.933629 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 21:00:22.971000 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 21:00:22.990710 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:00:23.019536 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:23.022361 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:00:23.029409 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20240502 21:00:23.105659 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240502 21:00:23.142086 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:23.284076 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:23.294420 27295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 21:00:23.295369 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:00:23.447571 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:23.645540 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:23.900892 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20240502 21:00:23.901281 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:23.998944 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:24.232311 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:00:24.415325 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:00:24.716176 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:00:24.795095 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:24.958909 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:00:25.177150 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20240502 21:00:25.271620 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:00:25.369103 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:00:25.387265 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20240502 21:00:25.609200 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:25.652458 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:25.687723 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:00:25.745890 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:25.756670 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20240502 21:00:25.833391 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 21:00:26.016767 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20240502 21:00:26.152071 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:00:26.253502 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
I20240502 21:00:26.288157 26768 raft_consensus.cc:482] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:00:26.288368 26768 raft_consensus.cc:504] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" member_type: VOTER last_known_addr { host: "127.19.66.132" port: 41007 } } peers { permanent_uuid: "d1f3f8e444364d6faa42d651b5dbb997" member_type: VOTER last_known_addr { host: "127.19.66.130" port: 43307 } attrs { replace: true } } peers { permanent_uuid: "1f40a11333e44ea0a34b6bc7618af522" member_type: VOTER last_known_addr { host: "127.19.66.133" port: 45949 } attrs { promote: false } } peers { permanent_uuid: "1af1e87d6ff2448d9502de9d7c2b8ef3" member_type: VOTER last_known_addr { host: "127.19.66.134" port: 46737 } attrs { promote: false } }
I20240502 21:00:26.289134 26768 leader_election.cc:288] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1f3f8e444364d6faa42d651b5dbb997 (127.19.66.130:43307), 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949), 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737)
I20240502 21:00:26.289829 27090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367543df62e048268908eb0d84c14876" candidate_uuid: "ebd8300729ec4226acc38b9cff2d8d3b" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1f3f8e444364d6faa42d651b5dbb997" is_pre_election: true
I20240502 21:00:26.290024 27090 raft_consensus.cc:2454] T 367543df62e048268908eb0d84c14876 P d1f3f8e444364d6faa42d651b5dbb997 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd8300729ec4226acc38b9cff2d8d3b in term 2.
W20240502 21:00:26.290921 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111)
W20240502 21:00:26.291141 27295 leader_election.cc:334] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111)
I20240502 21:00:26.291297 27295 leader_election.cc:302] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: d1f3f8e444364d6faa42d651b5dbb997, ebd8300729ec4226acc38b9cff2d8d3b; no voters: 1af1e87d6ff2448d9502de9d7c2b8ef3, 1f40a11333e44ea0a34b6bc7618af522
I20240502 21:00:26.291500 26768 raft_consensus.cc:2726] T 367543df62e048268908eb0d84c14876 P ebd8300729ec4226acc38b9cff2d8d3b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 21:00:26.303627 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20240502 21:00:26.310250 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20240502 21:00:26.706189 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:00:26.775046 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:26.858028 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 21:00:27.110481 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:27.345757 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:00:27.430150 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
0x00007fd9f775cd2d in __pthread_timedjoin_ex () from /lib/x86_64-linux-gnu/libpthread.so.0
W20240502 21:00:27.830483 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
  Id   Target Id         Frame 
* 1    Thread 0x7fd9eb38f480 (LWP 19722) "rebalancer_tool" 0x00007fd9f775cd2d in __pthread_timedjoin_ex () from /lib/x86_64-linux-gnu/libpthread.so.0
  2    Thread 0x7fd9e936a700 (LWP 19725) "rebalancer_tool" 0x00007fd9f776632a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
  3    Thread 0x7fd9e798d700 (LWP 19726) "kernel-watcher-" 0x00007fd9f7761fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  4    Thread 0x7fd9e618a700 (LWP 26769) "rpc reactor-267" 0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
W20240502 21:00:27.881330 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
  5    Thread 0x7fd9e698b700 (LWP 26770) "rpc reactor-267" 0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
  6    Thread 0x7fd9e5188700 (LWP 26771) "rpc reactor-267" 0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
  7    Thread 0x7fd9e5989700 (LWP 26772) "rpc reactor-267" 0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
  8    Thread 0x7fd9e718c700 (LWP 26773) "rpc reactor-267" 0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
  9    Thread 0x7fd9e4987700 (LWP 27951) "rpc reactor-279" 0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
  10   Thread 0x7fd9e3f7f700 (LWP 27952) "rpc reactor-279" 0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
  11   Thread 0x7fd9e377e700 (LWP 27953) "rpc reactor-279" 0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
  12   Thread 0x7fd9e2f7d700 (LWP 27954) "rpc reactor-279" 0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
  13   Thread 0x7fd9e1f7b700 (LWP 29302) "rebalancer_tool" 0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6

Thread 13 (Thread 0x7fd9e1f7b700 (LWP 29302)):
#0  0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd9f1091de0 in epoll_poll (loop=0x55f1bb1efa80, timeout=59.743000000000002) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
W20240502 21:00:28.017604 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20240502 21:00:28.018913 27159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:00:28.135253 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:00:28.141395 27026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240502 21:00:28.142338 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:00:28.154397 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:28.208359 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
#2  0x00007fd9f109549e in ev_run (loop=0x55f1bb1efa80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
W20240502 21:00:28.297750 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 21:00:28.298461 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
#3  0x00007fd9f70c2841 in ev::loop_ref::run (this=0x7fd9e1f7a228, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4  0x00007fd9f6a9ddc6 in kudu::(anonymous namespace)::ReadFdsFully (progname="/tmp/dist-test-taskF51BWD/build/debug/bin/kudu", fds=std::vector of length 2, capacity 2 = {...}, out=0x7fd9e1f7a370) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:240
W20240502 21:00:28.575913 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20240502 21:00:28.597380 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 21:00:28.698916 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20240502 21:00:28.702286 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20240502 21:00:28.748792 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20240502 21:00:28.792248 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20240502 21:00:28.997336 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:00:29.109521 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20240502 21:00:29.332590 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
#5  0x00007fd9f6aa1f36 in kudu::Subprocess::Call (argv=std::vector of length 12, capacity 16 = {...}, stdin_in="", stdout_out=0x7fd9e1f7a690, stderr_out=0x7fd9e1f7a6b0, env_vars=std::map with 0 elements) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:773
#6  0x00007fd9f8991c27 in kudu::tools::RunKuduTool (args=std::vector of length 3, capacity 3 = {...}, out=0x7fd9e1f7a690, err=0x7fd9e1f7a6b0, in="", env_vars=std::map with 0 elements) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/tool_test_util.cc:87
#7  0x000055f1b9a582de in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::<lambda()>::operator()(void) const (__closure=0x7ffe5f2d96c8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1255
#8  0x000055f1b9a5850b in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::<lambda()>::operator()(void) const (__closure=0x55f1bb594be8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1274
#9  0x000055f1b9a71438 in std::__invoke_impl<void, kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::<lambda()> >(std::__invoke_other, kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::<lambda()> &&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#10 0x000055f1b9a71086 in std::__invoke<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::<lambda()> >(kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::<lambda()> &&) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#11 0x000055f1b9a75c00 in std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x55f1bb594be8) at /usr/include/c++/7/thread:234
#12 0x000055f1b9a75ab1 in std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::<lambda()> > >::operator()(void) (this=0x55f1bb594be8) at /usr/include/c++/7/thread:243
#13 0x000055f1b9a75990 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::<lambda()> > > >::_M_run(void) (this=0x55f1bb594be0) at /usr/include/c++/7/thread:186
#14 0x00007fd9f57626df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
W20240502 21:00:29.659562 27158 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
#15 0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 12 (Thread 0x7fd9e2f7d700 (LWP 27954)):
#0  0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd9f1091de0 in epoll_poll (loop=0x55f1bb1edb00, timeout=0.099230412997712847) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2  0x00007fd9f109549e in ev_run (loop=0x55f1bb1edb00, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3  0x00007fd9f70c2841 in ev::loop_ref::run (this=0x55f1bb17fc60, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4  0x00007fd9f70bd31d in kudu::rpc::ReactorThread::RunThread (this=0x55f1bb17fc58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5  0x00007fd9f70b9c49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x55f1bb39cc58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
W20240502 21:00:29.791255 27025 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:00:30.019841 27297 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 21:00:30.160964 27024 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20240502 21:00:30.299100 27158 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
#6  0x00007fd9f70c1934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
W20240502 21:00:30.465608 26891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 21:00:30.500537 27296 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:00:30.563596 27160 consensus_peers.cc:485] T 078a1591d9d74e31a0407aeab76911b4 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
#7  0x00007fd9f87b8856 in std::function<void ()>::operator()() const (this=0x55f1bb39cc58) at /usr/include/c++/7/bits/std_function.h:706
#8  0x00007fd9f6aab807 in kudu::Thread::SuperviseThread (arg=0x55f1bb39cc00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9  0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 11 (Thread 0x7fd9e377e700 (LWP 27953)):
#0  0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd9f1091de0 in epoll_poll (loop=0x55f1bb1ed200, timeout=0.099633814996195724) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2  0x00007fd9f109549e in ev_run (loop=0x55f1bb1ed200, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3  0x00007fd9f70c2841 in ev::loop_ref::run (this=0x55f1bb17faa0, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4  0x00007fd9f70bd31d in kudu::rpc::ReactorThread::RunThread (this=0x55f1bb17fa98) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5  0x00007fd9f70b9c49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x55f1bb39ca58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6  0x00007fd9f70c1934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7  0x00007fd9f87b8856 in std::function<void ()>::operator()() const (this=0x55f1bb39ca58) at /usr/include/c++/7/bits/std_function.h:706
#8  0x00007fd9f6aab807 in kudu::Thread::SuperviseThread (arg=0x55f1bb39ca00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9  0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 10 (Thread 0x7fd9e3f7f700 (LWP 27952)):
#0  0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd9f1091de0 in epoll_poll (loop=0x55f1bb1ec000, timeout=0.099434672996721929) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2  0x00007fd9f109549e in ev_run (loop=0x55f1bb1ec000, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3  0x00007fd9f70c2841 in ev::loop_ref::run (this=0x55f1bb17fe20, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4  0x00007fd9f70bd31d in kudu::rpc::ReactorThread::RunThread (this=0x55f1bb17fe18) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5  0x00007fd9f70b9c49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x55f1bb39c958) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6  0x00007fd9f70c1934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7  0x00007fd9f87b8856 in std::function<void ()>::operator()() const (this=0x55f1bb39c958) at /usr/include/c++/7/bits/std_function.h:706
#8  0x00007fd9f6aab807 in kudu::Thread::SuperviseThread (arg=0x55f1bb39c900) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9  0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 9 (Thread 0x7fd9e4987700 (LWP 27951)):
#0  0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd9f1091de0 in epoll_poll (loop=0x55f1bb2ad600, timeout=0.099372379998385441) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2  0x00007fd9f109549e in ev_run (loop=0x55f1bb2ad600, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3  0x00007fd9f70c2841 in ev::loop_ref::run (this=0x55f1bb17f560, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4  0x00007fd9f70bd31d in kudu::rpc::ReactorThread::RunThread (this=0x55f1bb17f558) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5  0x00007fd9f70b9c49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x55f1bb3cee58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6  0x00007fd9f70c1934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7  0x00007fd9f87b8856 in std::function<void ()>::operator()() const (this=0x55f1bb3cee58) at /usr/include/c++/7/bits/std_function.h:706
#8  0x00007fd9f6aab807 in kudu::Thread::SuperviseThread (arg=0x55f1bb3cee00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9  0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 8 (Thread 0x7fd9e718c700 (LWP 26773)):
#0  0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd9f1091de0 in epoll_poll (loop=0x55f1bb2aa900, timeout=0.099058095966029214) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2  0x00007fd9f109549e in ev_run (loop=0x55f1bb2aa900, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3  0x00007fd9f70c2841 in ev::loop_ref::run (this=0x55f1bb17f3a0, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4  0x00007fd9f70bd31d in kudu::rpc::ReactorThread::RunThread (this=0x55f1bb17f398) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5  0x00007fd9f70b9c49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x55f1bb3cf058) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6  0x00007fd9f70c1934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7  0x00007fd9f87b8856 in std::function<void ()>::operator()() const (this=0x55f1bb3cf058) at /usr/include/c++/7/bits/std_function.h:706
#8  0x00007fd9f6aab807 in kudu::Thread::SuperviseThread (arg=0x55f1bb3cf000) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9  0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
W20240502 21:00:30.683495 27026 consensus_peers.cc:485] T d1b9ff3c921d474fa5ad106d1c17ab00 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
#10 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 7 (Thread 0x7fd9e5989700 (LWP 26772)):
#0  0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd9f1091de0 in epoll_poll (loop=0x55f1bb2aa480, timeout=0.099768034968292341) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2  0x00007fd9f109549e in ev_run (loop=0x55f1bb2aa480, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3  0x00007fd9f70c2841 in ev::loop_ref::run (this=0x55f1bb17f8e0, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4  0x00007fd9f70bd31d in kudu::rpc::ReactorThread::RunThread (this=0x55f1bb17f8d8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5  0x00007fd9f70b9c49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x55f1bb3cf858) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6  0x00007fd9f70c1934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7  0x00007fd9f87b8856 in std::function<void ()>::operator()() const (this=0x55f1bb3cf858) at /usr/include/c++/7/bits/std_function.h:706
#8  0x00007fd9f6aab807 in kudu::Thread::SuperviseThread (arg=0x55f1bb3cf800) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9  0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 6 (Thread 0x7fd9e5188700 (LWP 26771)):
#0  0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd9f1091de0 in epoll_poll (loop=0x55f1bb2aa000, timeout=0.099467039963201387) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2  0x00007fd9f109549e in ev_run (loop=0x55f1bb2aa000, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
W20240502 21:00:30.721645 27025 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
#3  0x00007fd9f70c2841 in ev::loop_ref::run (this=0x55f1bb17f720, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4  0x00007fd9f70bd31d in kudu::rpc::ReactorThread::RunThread (this=0x55f1bb17f718) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
W20240502 21:00:30.724434 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
#5  0x00007fd9f70b9c49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x55f1bb39c358) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6  0x00007fd9f70c1934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7  0x00007fd9f87b8856 in std::function<void ()>::operator()() const (this=0x55f1bb39c358) at /usr/include/c++/7/bits/std_function.h:706
#8  0x00007fd9f6aab807 in kudu::Thread::SuperviseThread (arg=0x55f1bb39c300) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9  0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 5 (Thread 0x7fd9e698b700 (LWP 26770)):
#0  0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd9f1091de0 in epoll_poll (loop=0x55f1bb2aad80, timeout=0.099858215966378339) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2  0x00007fd9f109549e in ev_run (loop=0x55f1bb2aad80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3  0x00007fd9f70c2841 in ev::loop_ref::run (this=0x55f1bb28dc60, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4  0x00007fd9f70bd31d in kudu::rpc::ReactorThread::RunThread (this=0x55f1bb28dc58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5  0x00007fd9f70b9c49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x55f1bb39ce58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6  0x00007fd9f70c1934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7  0x00007fd9f87b8856 in std::function<void ()>::operator()() const (this=0x55f1bb39ce58) at /usr/include/c++/7/bits/std_function.h:706
#8  0x00007fd9f6aab807 in kudu::Thread::SuperviseThread (arg=0x55f1bb39ce00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9  0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 4 (Thread 0x7fd9e618a700 (LWP 26769)):
#0  0x00007fd9f4e1fa47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd9f1091de0 in epoll_poll (loop=0x55f1bb2ada80, timeout=0.099870176964031998) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev_epoll.c:153
#2  0x00007fd9f109549e in ev_run (loop=0x55f1bb2ada80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/libev-4.20/ev.c:3521
#3  0x00007fd9f70c2841 in ev::loop_ref::run (this=0x55f1bb28daa0, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4  0x00007fd9f70bd31d in kudu::rpc::ReactorThread::RunThread (this=0x55f1bb28da98) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:510
#5  0x00007fd9f70b9c49 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x55f1bb39cb58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:197
#6  0x00007fd9f70c1934 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7  0x00007fd9f87b8856 in std::function<void ()>::operator()() const (this=0x55f1bb39cb58) at /usr/include/c++/7/bits/std_function.h:706
#8  0x00007fd9f6aab807 in kudu::Thread::SuperviseThread (arg=0x55f1bb39cb00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9  0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 3 (Thread 0x7fd9e798d700 (LWP 19726)):
#0  0x00007fd9f7761fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fd9f696d7eb in kudu::ConditionVariable::WaitUntil (this=0x55f1bb178438, until=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:87
#2  0x000055f1b9a77f5c in kudu::CountDownLatch::WaitUntil (this=0x55f1bb178400, when=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:87
#3  0x000055f1b9a78006 in kudu::CountDownLatch::WaitFor (this=0x55f1bb178400, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:97
#4  0x00007fd9f6a05166 in kudu::KernelStackWatchdog::RunThread (this=0x55f1bb178360) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/kernel_stack_watchdog.cc:134
#5  0x00007fd9f6a04375 in kudu::KernelStackWatchdog::<lambda()>::operator()(void) const (__closure=0x55f1bb04d358) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/kernel_stack_watchdog.cc:71
W20240502 21:00:30.925352 27297 consensus_peers.cc:485] T 6a8ce1df86914b2caf2d083d97eda4e6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
#6  0x00007fd9f6a05aca in std::_Function_handler<void(), kudu::KernelStackWatchdog::KernelStackWatchdog()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7  0x00007fd9f87b8856 in std::function<void ()>::operator()() const (this=0x55f1bb04d358) at /usr/include/c++/7/bits/std_function.h:706
#8  0x00007fd9f6aab807 in kudu::Thread::SuperviseThread (arg=0x55f1bb04d300) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9  0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x7fd9e936a700 (LWP 19725)):
#0  0x00007fd9f776632a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fd9f6aa27b5 in kudu::Subprocess::DoWait (this=0x7fd9e9369380, wait_status=0x0, mode=kudu::Subprocess::BLOCKING) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:817
#2  0x00007fd9f6a9fbe8 in kudu::Subprocess::Wait (this=0x7fd9e9369380, wait_status=0x0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:544
#3  0x00007fd9f69f2b7c in kudu::PstackWatcher::RunStackDump (argv=std::vector of length 12, capacity 16 = {...}) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:258
#4  0x00007fd9f69f26ab in kudu::PstackWatcher::RunGdbStackDump (pid=19722, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:218
#5  0x00007fd9f69f1f09 in kudu::PstackWatcher::DumpPidStacks (pid=19722, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:169
W20240502 21:00:31.070375 27024 consensus_peers.cc:485] T ff9fef57f6eb490da536170a02b96212 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20240502 21:00:31.075671 27024 consensus_peers.cc:485] T d44f271f166b4c288b76689a3e3764d3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1f40a11333e44ea0a34b6bc7618af522 (127.19.66.133:45949): Couldn't send request to peer 1f40a11333e44ea0a34b6bc7618af522. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.133:45949: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
#6  0x00007fd9f69f1e82 in kudu::PstackWatcher::DumpStacks (flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:162
#7  0x00007fd9f8979087 in kudu::<lambda()>::operator()(void) const (__closure=0x55f1bb0b1bf8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_main.cc:60
#8  0x00007fd9f8979888 in std::__invoke_impl<void, kudu::CreateAndStartTimeoutThread()::<lambda()> >(std::__invoke_other, kudu::<lambda()> &&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#9  0x00007fd9f89795dc in std::__invoke<kudu::CreateAndStartTimeoutThread()::<lambda()> >(kudu::<lambda()> &&) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#10 0x00007fd9f8979e6e in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x55f1bb0b1bf8) at /usr/include/c++/7/thread:234
#11 0x00007fd9f8979e13 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::operator()(void) (this=0x55f1bb0b1bf8) at /usr/include/c++/7/thread:243
#12 0x00007fd9f8979dce in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > > >::_M_run(void) (this=0x55f1bb0b1bf0) at /usr/include/c++/7/thread:186
#13 0x00007fd9f57626df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007fd9f775b6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007fd9f4e1f71f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7fd9eb38f480 (LWP 19722)):
#0  0x00007fd9f775cd2d in __pthread_timedjoin_ex () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fd9f5762933 in std::thread::join() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
W20240502 21:00:31.144207 27026 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20240502 21:00:31.203572 27159 consensus_peers.cc:485] T b5f81e46ab1f48fc8a88495b42f83640 P c107c43e538d48f58625c4320f0aa4b8 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20240502 21:00:31.249612 27295 consensus_peers.cc:485] T f00d12e425284c36ae8baed56c7cfaf6 P ebd8300729ec4226acc38b9cff2d8d3b -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 21:00:31.392580 27025 consensus_peers.cc:485] T 135a08f1f2ac4707bbacee81ac73a19e P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 6706d1c0946d42a9bae62d719e1e5c31 (127.19.66.136:41529): Couldn't send request to peer 6706d1c0946d42a9bae62d719e1e5c31. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.136:41529: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20240502 21:00:31.443578 26891 consensus_peers.cc:485] T a2b61909194d488682afd91c0f796405 P 297de823852d4a8da44791ab03131cd3 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
#2  0x000055f1b9a58a9c in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody (this=0x55f1bb06b000) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1281
#3  0x00007fd9f84f90ed in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7fd9f84fa82b "the test body", method=<optimized out>, object=0x55f1bb06b000) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#4  testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x55f1bb06b000, method=<optimized out>, location=location@entry=0x7fd9f84fa82b "the test body") at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#5  0x00007fd9f84edbdb in testing::Test::Run (this=0x55f1bb06b000) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#6  0x00007fd9f84edd9d in testing::TestInfo::Run (this=0x55f1bb114ea0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#7  0x00007fd9f84ee377 in testing::TestSuite::Run (this=0x55f1bb114d80) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#8  0x00007fd9f84ee77c in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870
#9  0x00007fd9f84f960d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7fd9f84fc3d8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x55f1bb0ec280) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#10 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55f1bb0ec280, method=<optimized out>, location=location@entry=0x7fd9f84fc3d8 "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#11 0x00007fd9f84ede63 in testing::UnitTest::Run (this=0x7fd9f870d1e0 <testing::UnitTest::GetInstance()::instance>) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444
#12 0x00007fd9f897a1de in RUN_ALL_TESTS () at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#13 0x00007fd9f897953d in main (argc=1, argv=0x7ffe5f2d9dd8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_main.cc:109
#14 0x00007fd9f4d1fbf7 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x000055f1b9a4313a in _start ()
W20240502 21:00:31.670675 27026 consensus_peers.cc:485] T 5704bdfe6c02437c8e9c8c3deaadb9e3 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1bc8a142cfc7487bacc573afb0a92b74 (127.19.66.135:46845): Couldn't send request to peer 1bc8a142cfc7487bacc573afb0a92b74. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.135:46845: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
************************* END STACKS ***************************
F20240502 21:00:31.695128 19725 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1714683631 (unix time) try "date -d @1714683631" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x3e800004d0a) received by PID 19722 (TID 0x7fd9e936a700) from PID 19722; stack trace: ***
    @     0x7fd9f7766980 (unknown) at ??:0
    @     0x7fd9f4d3cfb7 gsignal at ??:0
    @     0x7fd9f4d3e921 abort at ??:0
    @     0x7fd9f5f7cdcd google::LogMessage::Fail() at ??:0
    @     0x7fd9f5f80b93 google::LogMessage::SendToLog() at ??:0
    @     0x7fd9f5f7c7cc google::LogMessage::Flush() at ??:0
    @     0x7fd9f5f7df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7fd9f89791c4 _ZZN4kuduL27CreateAndStartTimeoutThreadEvENKUlvE_clEv at ??:0
    @     0x7fd9f8979888 _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ at ??:0
    @     0x7fd9f89795dc _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ at ??:0
    @     0x7fd9f8979e6e _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
    @     0x7fd9f8979e13 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEclEv at ??:0
    @     0x7fd9f8979dce _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEEE6_M_runEv at ??:0
    @     0x7fd9f57626df (unknown) at ??:0
    @     0x7fd9f775b6db start_thread at ??:0
    @     0x7fd9f4e1f71f clone at ??:0
W20240502 21:00:31.824416 27024 consensus_peers.cc:485] T a547561343304137a044ce11517f6007 P d1f3f8e444364d6faa42d651b5dbb997 -> Peer 1af1e87d6ff2448d9502de9d7c2b8ef3 (127.19.66.134:46737): Couldn't send request to peer 1af1e87d6ff2448d9502de9d7c2b8ef3. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.134:46737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.